Skip to contentSkip to conversation

xdp: hints via kfuncs

PATCH Superseded Stanislav Fomichev 11 participants 54 messages Nov 21, 2022 – Dec 1, 2022 Page 2 of 2

Subsystems (14)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#22 lore raw

PATCH v2 4/8 veth: Support RX XDP metadata

In reply to Stanislav Fomichev

The goal is to enable end-to-end testing of the metadata for AF_XDP.

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 drivers/net/veth.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
drivers/net/veth.c +32 -0
index bbabc592d431..fdbca2aee33a 100644
@@ -118,6 +118,7 @@ static struct {
 
 struct veth_xdp_buff {
 	struct xdp_buff xdp;
+	struct sk_buff *skb;
 };
 
 static int veth_get_link_ksettings(struct net_device *dev,
@@ -602,6 +603,7 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
 
 		xdp_convert_frame_to_buff(frame, xdp);
 		xdp->rxq = &rq->xdp_rxq;
+		vxbuf.skb = NULL;
 
 		act = bpf_prog_run_xdp(xdp_prog, xdp);
 
@@ -823,6 +825,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 	__skb_push(skb, skb->data - skb_mac_header(skb));
 	if (veth_convert_skb_to_xdp_buff(rq, xdp, &skb))
 		goto drop;
+	vxbuf.skb = skb;
 
 	orig_data = xdp->data;
 	orig_data_end = xdp->data_end;
@@ -1665,6 +1668,30 @@ static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp)
 	}
 }
 
+static bool veth_xdp_rx_timestamp_supported(const struct xdp_md *ctx)
+{
+	return true;
+}
+
+static u64 veth_xdp_rx_timestamp(const struct xdp_md *ctx)
+{
+	return ktime_get_mono_fast_ns();
+}
+
+static bool veth_xdp_rx_hash_supported(const struct xdp_md *ctx)
+{
+	return true;
+}
+
+static u32 veth_xdp_rx_hash(const struct xdp_md *ctx)
+{
+	struct veth_xdp_buff *_ctx = (void *)ctx;
+
+	if (_ctx->skb)
+		return skb_get_hash(_ctx->skb);
+	return 0;
+}
+
 static const struct net_device_ops veth_netdev_ops = {
 	.ndo_init            = veth_dev_init,
 	.ndo_open            = veth_open,
@@ -1684,6 +1711,11 @@ static const struct net_device_ops veth_netdev_ops = {
 	.ndo_bpf		= veth_xdp,
 	.ndo_xdp_xmit		= veth_ndo_xdp_xmit,
 	.ndo_get_peer_dev	= veth_peer_dev,
+
+	.ndo_xdp_rx_timestamp_supported = veth_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp	= veth_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = veth_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash	= veth_xdp_rx_hash,
 };
 
 #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#23 lore raw

PATCH v2 5/8 selftests/bpf: Verify xdp_metadata xdp->af_xdp path

In reply to Stanislav Fomichev

- create new netns
- create veth pair (veTX+veRX)
- setup AF_XDP socket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRX

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../selftests/bpf/prog_tests/xdp_metadata.c   | 365 ++++++++++++++++++
 .../selftests/bpf/progs/xdp_metadata.c        |  57 +++
 tools/testing/selftests/bpf/xdp_metadata.h    |   7 +
 4 files changed, 430 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_metadata.c
 create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata.c
 create mode 100644 tools/testing/selftests/bpf/xdp_metadata.h
tools/testing/selftests/bpf/Makefile +1 -1
index 6a0f043dc410..4eed22fa3681 100644
@@ -527,7 +527,7 @@ TRUNNER_BPF_PROGS_DIR := progs
 TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c	\
 			 network_helpers.c testing_helpers.c		\
 			 btf_helpers.c flow_dissector_load.h		\
-			 cap_helpers.c
+			 cap_helpers.c xsk.c
 TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/liburandom_read.so			\
 		       $(OUTPUT)/xdp_synproxy				\
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c new file +365 -0

365 lines not on this page · raw message

tools/testing/selftests/bpf/progs/xdp_metadata.c new file +57 -0

57 lines not on this page · raw message

tools/testing/selftests/bpf/xdp_metadata.h new file +7 -0
new file mode 100644
index 000000000000..c4892d122b7f
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#pragma once
+
+struct xdp_meta {
+	__u64 rx_timestamp;
+	__u32 rx_hash;
+};
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#24 lore raw

Re: [PATCH bpf-next v2 5/8] selftests/bpf: Verify xdp_metadata xdp->af_xdp path

tools/testing/selftests/bpf/progs/xdp_metadata.c line 52

Quoted 9 lines · Stanislav Fomichev

[...]

tools/testing/selftests/bpf/progs/xdp_metadata.c new file +6 -0
+
+	if (bpf_xdp_metadata_rx_timestamp_supported(ctx))
+		meta->rx_timestamp = bpf_xdp_metadata_rx_timestamp(ctx);
+
+	if (bpf_xdp_metadata_rx_hash_supported(ctx))
+		meta->rx_hash = bpf_xdp_metadata_rx_hash(ctx);

Is there a case when F_supported and F are not called in a sequence? If not, then you can join them:

	bool (*ndo_xdp_rx_timestamp)(const struct xdp_md *ctx, u64 *timestamp);

so that a calling XDP program does one indirect call instead of two for one field

	if (bpf_xdp_metadata_rx_timestamp(ctx, &meta->rx_timestamp)) {
		/* ... couldn't get the timestamp */
	}
#25 lore raw

Re: [PATCH bpf-next v2 5/8] selftests/bpf: Verify xdp_metadata xdp->af_xdp path

Quoted 15 lines · Anton Protopopov

tools/testing/selftests/bpf/progs/xdp_metadata.c line 52

Quoted 9 lines · Stanislav Fomichev

[...]

tools/testing/selftests/bpf/progs/xdp_metadata.c new file +6 -0
+
+     if (bpf_xdp_metadata_rx_timestamp_supported(ctx))
+             meta->rx_timestamp = bpf_xdp_metadata_rx_timestamp(ctx);
+
+     if (bpf_xdp_metadata_rx_hash_supported(ctx))
+             meta->rx_hash = bpf_xdp_metadata_rx_hash(ctx);

Is there a case when F_supported and F are not called in a sequence? If not, then you can join them:

Quoted 8 lines
        bool (*ndo_xdp_rx_timestamp)(const struct xdp_md *ctx, u64 *timestamp);

so that a calling XDP program does one indirect call instead of two for one field

        if (bpf_xdp_metadata_rx_timestamp(ctx, &meta->rx_timestamp)) {
                /* ... couldn't get the timestamp */
        }

The purpose of the original bpf_xdp_metadata_rx_hash_supported was to allow unrolling and support dropping some dead branches by the verifier.
Since there is still a chance we might eventually unroll some of these, maybe it makes sense to keep as is?

#26 lore raw

PATCH v2 6/8 mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

In reply to Stanislav Fomichev

No functional changes. Boilerplate to allow stuffing more data after xdp_buff.

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++---------
 1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +15 -11
index 8f762fc170b3..467356633172 100644
@@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
 #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)
 #endif
 
+struct mlx4_xdp_buff {
+	struct xdp_buff xdp;
+};
+
 int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)
 {
 	struct mlx4_en_priv *priv = netdev_priv(dev);
 	int factor = priv->cqe_factor;
 	struct mlx4_en_rx_ring *ring;
+	struct mlx4_xdp_buff mxbuf;
 	struct bpf_prog *xdp_prog;
 	int cq_ring = cq->ring;
 	bool doorbell_pending;
 	bool xdp_redir_flush;
 	struct mlx4_cqe *cqe;
-	struct xdp_buff xdp;
 	int polled = 0;
 	int index;
 
@@ -681,7 +685,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
 	ring = priv->rx_ring[cq_ring];
 
 	xdp_prog = rcu_dereference_bh(ring->xdp_prog);
-	xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
+	xdp_init_buff(&mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
 	doorbell_pending = false;
 	xdp_redir_flush = false;
 
@@ -776,24 +780,24 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
 						priv->frag_info[0].frag_size,
 						DMA_FROM_DEVICE);
 
-			xdp_prepare_buff(&xdp, va - frags[0].page_offset,
+			xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,
 					 frags[0].page_offset, length, false);
-			orig_data = xdp.data;
+			orig_data = mxbuf.xdp.data;
 
-			act = bpf_prog_run_xdp(xdp_prog, &xdp);
+			act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);
 
-			length = xdp.data_end - xdp.data;
-			if (xdp.data != orig_data) {
-				frags[0].page_offset = xdp.data -
-					xdp.data_hard_start;
-				va = xdp.data;
+			length = mxbuf.xdp.data_end - mxbuf.xdp.data;
+			if (mxbuf.xdp.data != orig_data) {
+				frags[0].page_offset = mxbuf.xdp.data -
+					mxbuf.xdp.data_hard_start;
+				va = mxbuf.xdp.data;
 			}
 
 			switch (act) {
 			case XDP_PASS:
 				break;
 			case XDP_REDIRECT:
-				if (likely(!xdp_do_redirect(dev, &xdp, xdp_prog))) {
+				if (likely(!xdp_do_redirect(dev, &mxbuf.xdp, xdp_prog))) {
 					ring->xdp_redirect++;
 					xdp_redir_flush = true;
 					frags[0].page = NULL;
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#27 lore raw

Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 28 lines · Stanislav Fomichev

No functional changes. Boilerplate to allow stuffing more data after xdp_buff.

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
  drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++---------
  1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +0 -0
index 8f762fc170b3..467356633172 100644
@@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
  #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)
  #endif
 

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, · line 673

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +5 -0
+struct mlx4_xdp_buff {
+	struct xdp_buff xdp;
+};
+
  int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)
  {
  	struct mlx4_en_priv *priv = netdev_priv(dev);
  	int factor = priv->cqe_factor;
  	struct mlx4_en_rx_ring *ring;
+	struct mlx4_xdp_buff this would helpmxbuf;

as it doesn't go through an init function (only mxbuf.xdp does), better init to zero.

Quoted 52 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +10 -11
  	struct bpf_prog *xdp_prog;
  	int cq_ring = cq->ring;
  	bool doorbell_pending;
  	bool xdp_redir_flush;
  	struct mlx4_cqe *cqe;
-	struct xdp_buff xdp;
  	int polled = 0;
  	int index;
 
@@ -681,7 +685,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
  	ring = priv->rx_ring[cq_ring];
 
  	xdp_prog = rcu_dereference_bh(ring->xdp_prog);
-	xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
+	xdp_init_buff(&mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
  	doorbell_pending = false;
  	xdp_redir_flush = false;
 
@@ -776,24 +780,24 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
  						priv->frag_info[0].frag_size,
  						DMA_FROM_DEVICE);
 
-			xdp_prepare_buff(&xdp, va - frags[0].page_offset,
+			xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,
  					 frags[0].page_offset, length, false);
-			orig_data = xdp.data;
+			orig_data = mxbuf.xdp.data;
 
-			act = bpf_prog_run_xdp(xdp_prog, &xdp);
+			act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);
 
-			length = xdp.data_end - xdp.data;
-			if (xdp.data != orig_data) {
-				frags[0].page_offset = xdp.data -
-					xdp.data_hard_start;
-				va = xdp.data;
+			length = mxbuf.xdp.data_end - mxbuf.xdp.data;
+			if (mxbuf.xdp.data != orig_data) {
+				frags[0].page_offset = mxbuf.xdp.data -
+					mxbuf.xdp.data_hard_start;
+				va = mxbuf.xdp.data;
  			}
 
  			switch (act) {
  			case XDP_PASS:
  				break;
  			case XDP_REDIRECT:
-				if (likely(!xdp_do_redirect(dev, &xdp, xdp_prog))) {
+				if (likely(!xdp_do_redirect(dev, &mxbuf.xdp, xdp_prog))) {
  					ring->xdp_redirect++;
  					xdp_redir_flush = true;
  					frags[0].page = NULL;
#28 lore raw

Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 32 lines · Tariq Toukan
Quoted 28 lines · Stanislav Fomichev

No functional changes. Boilerplate to allow stuffing more data after xdp_buff.

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
  drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++---------
  1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +0 -0
index 8f762fc170b3..467356633172 100644
@@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
  #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)
  #endif
 
Quoted 13 lines

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +5 -0
+struct mlx4_xdp_buff {
+     struct xdp_buff xdp;
+};
+
  int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)
  {
      struct mlx4_en_priv *priv = netdev_priv(dev);
      int factor = priv->cqe_factor;
      struct mlx4_en_rx_ring *ring;
+     struct mlx4_xdp_buff this would helpmxbuf;

as it doesn't go through an init function (only mxbuf.xdp does), better init to zero.

SG, will do, thanks!

Quoted 52 lines
Quoted 52 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +10 -11
      struct bpf_prog *xdp_prog;
      int cq_ring = cq->ring;
      bool doorbell_pending;
      bool xdp_redir_flush;
      struct mlx4_cqe *cqe;
-     struct xdp_buff xdp;
      int polled = 0;
      int index;
 
@@ -681,7 +685,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
      ring = priv->rx_ring[cq_ring];
 
      xdp_prog = rcu_dereference_bh(ring->xdp_prog);
-     xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
+     xdp_init_buff(&mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
      doorbell_pending = false;
      xdp_redir_flush = false;
 
@@ -776,24 +780,24 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
                                              priv->frag_info[0].frag_size,
                                              DMA_FROM_DEVICE);
 
-                     xdp_prepare_buff(&xdp, va - frags[0].page_offset,
+                     xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,
                                       frags[0].page_offset, length, false);
-                     orig_data = xdp.data;
+                     orig_data = mxbuf.xdp.data;
 
-                     act = bpf_prog_run_xdp(xdp_prog, &xdp);
+                     act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);
 
-                     length = xdp.data_end - xdp.data;
-                     if (xdp.data != orig_data) {
-                             frags[0].page_offset = xdp.data -
-                                     xdp.data_hard_start;
-                             va = xdp.data;
+                     length = mxbuf.xdp.data_end - mxbuf.xdp.data;
+                     if (mxbuf.xdp.data != orig_data) {
+                             frags[0].page_offset = mxbuf.xdp.data -
+                                     mxbuf.xdp.data_hard_start;
+                             va = mxbuf.xdp.data;
                      }
 
                      switch (act) {
                      case XDP_PASS:
                              break;
                      case XDP_REDIRECT:
-                             if (likely(!xdp_do_redirect(dev, &xdp, xdp_prog))) {
+                             if (likely(!xdp_do_redirect(dev, &mxbuf.xdp, xdp_prog))) {
                                      ring->xdp_redirect++;
                                      xdp_redir_flush = true;
                                      frags[0].page = NULL;
#29 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

In reply to Stanislav Fomichev
Quoted 25 lines · Stanislav Fomichev

No functional changes. Boilerplate to allow stuffing more data after xdp_buff.

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++---------
 1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +0 -0
index 8f762fc170b3..467356633172 100644

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, · line 666

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +3 -0
@@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
 #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)
 #endif
 
+struct mlx4_xdp_buff {
+	struct xdp_buff xdp;
+};

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing there; and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a full cache line anyway, so any data stuffed after it will spill over to a new one; so I don't think there's much difference performance-wise.

I'll send my patch to add support to mlx5 (using the drv_priv pointer approach) separately.

-Toke

#30 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 31 lines · Toke Høiland-Jørgensen
Quoted 28 lines · Stanislav Fomichev

No functional changes. Boilerplate to allow stuffing more data after xdp_buff.

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++---------
 1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +0 -0
index 8f762fc170b3..467356633172 100644
@@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
 #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)
 #endif
 
Quoted 12 lines

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, · line 666

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +3 -0
+struct mlx4_xdp_buff {
+     struct xdp_buff xdp;
+};

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing there; and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a full cache line anyway, so any data stuffed after it will spill over to a new one; so I don't think there's much difference performance-wise.

I guess the alternative is to extend xsk_buff_pool with some new argument for xdp_buff tailroom? (so it can kmalloc(sizeof(xdp_buff) + xdp_buff_tailroom))
But it seems messy because there is no way of knowing what the target device's tailroom is, so it has to be a user setting :-/ I've started with a priv pointer in xdp_buff initially, it seems fine to go back. I'll probably convert veth/mlx4 to the same mode as well to avoid having different approaches in different places..

Quoted 2 lines

I'll send my patch to add support to mlx5 (using the drv_priv pointer approach) separately.

Saw them, thanks! Will include them in v3+.

maintainer
#31 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 9 lines · On Wed, 23 Nov 2022 10:26:41 -0800 Stanislav Fomichev wrote:
Quoted 8 lines

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing there; and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a full cache line anyway, so any data stuffed after it will spill over to a new one; so I don't think there's much difference performance-wise.

Quoted 8 lines

I guess the alternative is to extend xsk_buff_pool with some new argument for xdp_buff tailroom? (so it can kmalloc(sizeof(xdp_buff) + xdp_buff_tailroom))
But it seems messy because there is no way of knowing what the target device's tailroom is, so it has to be a user setting :-/ I've started with a priv pointer in xdp_buff initially, it seems fine to go back. I'll probably convert veth/mlx4 to the same mode as well to avoid having different approaches in different places..

Can we not do this please? Add 16B of "private driver space" after the xdp_buff in xdp_buff_xsk (we have 16B to full cacheline), the drivers decide how they use it. Drivers can do BUILD_BUG_ON() for their expected size and cast that to whatever struct they want. This is how various offloads work, the variable size tailroom would be an over design IMO.

And this way non XSK paths can keep its normal typing.

Quoted 4 lines
Quoted 2 lines

I'll send my patch to add support to mlx5 (using the drv_priv pointer approach) separately.

Saw them, thanks! Will include them in v3+.

#32 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 22 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 23 Nov 2022 10:26:41 -0800 Stanislav Fomichev wrote:
Quoted 3 lines

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing

there;

Quoted 3 lines
Quoted 3 lines

and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a

full

Quoted 1 line
Quoted 1 line

cache line anyway, so any data stuffed after it will spill over to a

new

Quoted 10 lines
Quoted 1 line

one; so I don't think there's much difference performance-wise.

I guess the alternative is to extend xsk_buff_pool with some new argument for xdp_buff tailroom? (so it can kmalloc(sizeof(xdp_buff) + xdp_buff_tailroom))
But it seems messy because there is no way of knowing what the target device's tailroom is, so it has to be a user setting :-/ I've started with a priv pointer in xdp_buff initially, it seems fine to go back. I'll probably convert veth/mlx4 to the same mode as well to avoid having different approaches in different places..

Quoted 8 lines

Can we not do this please? Add 16B of "private driver space" after the xdp_buff in xdp_buff_xsk (we have 16B to full cacheline), the drivers decide how they use it. Drivers can do BUILD_BUG_ON() for their expected size and cast that to whatever struct they want. This is how various offloads work, the variable size tailroom would be an over design IMO.

And this way non XSK paths can keep its normal typing.

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */

/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
dma_addr_t dma; /* 72 8 */
dma_addr_t frame_dma; /* 80 8 */
struct xsk_buff_pool * pool; /* 88 8 */
u64 orig_addr; /* 96 8 */
struct list_head free_list_node; /* 104 16 */

/* size: 120, cachelines: 2, members: 7 */ /* last cacheline: 56 bytes */
};

Toke, I can try to merge this into your patch + keep your SoB (or feel free to try this and retest yourself, whatever works).

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +5 -0
index bc2d9034af5b..837bf103b871 100644
@@ -44,6 +44,11 @@
 	(MLX5E_XDP_INLINE_WQE_MAX_DS_CNT * MLX5_SEND_WQE_DS - \
 	 sizeof(struct mlx5_wqe_inline_seg))
 
+struct mlx5_xdp_cb {
+	struct mlx5_cqe64 *cqe;
+	struct mlx5e_rq *rq;
+};
+
 struct mlx5e_xsk_param;
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param  

*xsk);
bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct page *page,

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +7 -0
index c91b54d9ff27..84d23b2da7ce 100644
@@ -5,6 +5,7 @@
 #include "en/xdp.h"
 #include <net/xdp_sock_drv.h>
 #include <linux/filter.h>
+#include <linux/build_bug.h>
 
 /* RX data path */
 
@@ -286,8 +287,14 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
 					      u32 cqe_bcnt)
 {
 	struct xdp_buff *xdp = wi->au->xsk;
+	struct mlx5_xdp_cb *cb;
 	struct bpf_prog *prog;
 
+	BUILD_BUG_ON(sizeof(struct mlx5_xdp_cb) > XSKB_CB_SIZE);
+	cb = xp_get_cb(xdp);
+	cb->cqe = NULL /*cqe*/;
+	cb->rq = rq;
+
 	/* wi->offset is not used in this function, because xdp->data and the
 	 * DMA address point directly to the necessary place. Furthermore, the
 	 * XSK allocator allocates frames per packet, instead of pages, so
include/net/xsk_buff_pool.h +8 -0
index f787c3f524b0..b298590429e7 100644
@@ -19,8 +19,11 @@ struct xdp_sock;
 struct device;
 struct page;
 
+#define XSKB_CB_SIZE 16
+
 struct xdp_buff_xsk {
 	struct xdp_buff xdp;
+	u8 cb[XSKB_CB_SIZE]; /* Private area for the drivers to use. */
 	dma_addr_t dma;
 	dma_addr_t frame_dma;
 	struct xsk_buff_pool *pool;
@@ -143,6 +146,11 @@ static inline dma_addr_t xp_get_frame_dma(struct xdp_buff_xsk *xskb)
 	return xskb->frame_dma;
 }
 
+static inline void *xp_get_cb(struct xdp_buff *xdp)
+{
+	return (void *)xdp + offsetof(struct xdp_buff_xsk, cb);
+}
+
 void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb);
 static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb)
 {
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

I'll send my patch to add support to mlx5 (using the drv_priv pointer approach) separately.

Saw them, thanks! Will include them in v3+.

#33 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 23 lines · On Wed, Nov 23, 2022 at 11:52:12AM -0800, [email protected] wrote:
Quoted 21 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 23 Nov 2022 10:26:41 -0800 Stanislav Fomichev wrote:
Quoted 3 lines

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing

there;

Quoted 3 lines
Quoted 3 lines

and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a

full

Quoted 1 line
Quoted 1 line

cache line anyway, so any data stuffed after it will spill over to a

new

Quoted 10 lines
Quoted 1 line

one; so I don't think there's much difference performance-wise.

I guess the alternative is to extend xsk_buff_pool with some new argument for xdp_buff tailroom? (so it can kmalloc(sizeof(xdp_buff) + xdp_buff_tailroom))
But it seems messy because there is no way of knowing what the target device's tailroom is, so it has to be a user setting :-/ I've started with a priv pointer in xdp_buff initially, it seems fine to go back. I'll probably convert veth/mlx4 to the same mode as well to avoid having different approaches in different places..

Quoted 2 lines
Quoted 2 lines

Can we not do this please? Add 16B of "private driver space" after the xdp_buff in xdp_buff_xsk (we have 16B to full cacheline), the

It is time to jump the hints train I guess:D

We have 8 bytes left in the cacheline that xdp_buff occupies - pahole output below shows that cb spans through two cachelines. Did you mean something else though?

Quoted 59 lines
Quoted 6 lines

drivers decide how they use it. Drivers can do BUILD_BUG_ON() for their expected size and cast that to whatever struct they want. This is how various offloads work, the variable size tailroom would be an over design IMO.

And this way non XSK paths can keep its normal typing.

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */

/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
dma_addr_t dma; /* 72 8 */
dma_addr_t frame_dma; /* 80 8 */
struct xsk_buff_pool * pool; /* 88 8 */
u64 orig_addr; /* 96 8 */
struct list_head free_list_node; /* 104 16 */

/* size: 120, cachelines: 2, members: 7 */ /* last cacheline: 56 bytes */
};

Toke, I can try to merge this into your patch + keep your SoB (or feel free to try this and retest yourself, whatever works).

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +5 -0
index bc2d9034af5b..837bf103b871 100644
@@ -44,6 +44,11 @@
 	(MLX5E_XDP_INLINE_WQE_MAX_DS_CNT * MLX5_SEND_WQE_DS - \
 	 sizeof(struct mlx5_wqe_inline_seg))
 
+struct mlx5_xdp_cb {
+	struct mlx5_cqe64 *cqe;
+	struct mlx5e_rq *rq;
+};
+
 struct mlx5e_xsk_param;
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param
 *xsk);

bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct page *page,

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
index c91b54d9ff27..84d23b2da7ce 100644
@@ -5,6 +5,7 @@
 #include "en/xdp.h"
 #include <net/xdp_sock_drv.h>
 #include <linux/filter.h>
+#include <linux/build_bug.h>
 
 /* RX data path */
 
@@ -286,8 +287,14 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct
 mlx5e_rq *rq,

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct · line 297

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +5 -0
 					      u32 cqe_bcnt)
 {
 	struct xdp_buff *xdp = wi->au->xsk;
+	struct mlx5_xdp_cb *cb;
 	struct bpf_prog *prog;
 
+	BUILD_BUG_ON(sizeof(struct mlx5_xdp_cb) > XSKB_CB_SIZE);
+	cb = xp_get_cb(xdp);
+	cb->cqe = NULL /*cqe*/;
+	cb->rq = rq;

I believe that these could be set once at a setup time within a pool - take a look at xsk_pool_set_rxq_info(). This will save us cycles so that we will skip assignments per each processed xdp_buff.

AF_XDP ZC performance comes in a major part from the fact that thanks to xsk_buff_pool we have less work to do per each processed buffer.

Quoted 21 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
+
 	/* wi->offset is not used in this function, because xdp->data and the
 	 * DMA address point directly to the necessary place. Furthermore, the
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +0 -0
 	 * XSK allocator allocates frames per packet, instead of pages, so
include/net/xsk_buff_pool.h +3 -0
index f787c3f524b0..b298590429e7 100644
@@ -19,8 +19,11 @@ struct xdp_sock;
 struct device;
 struct page;
 
+#define XSKB_CB_SIZE 16
+
 struct xdp_buff_xsk {
 	struct xdp_buff xdp;
+	u8 cb[XSKB_CB_SIZE]; /* Private area for the drivers to use. */
 	dma_addr_t dma;
 	dma_addr_t frame_dma;
 	struct xsk_buff_pool *pool;

include/net/xsk_buff_pool.h static inline dma_addr_t xp_get_frame_dma(struct · line 153

Quoted 8 lines
include/net/xsk_buff_pool.h +4 -0
@@ -143,6 +146,11 @@ static inline dma_addr_t xp_get_frame_dma(struct
 xdp_buff_xsk *xskb)
 	return xskb->frame_dma;
 }
 
+static inline void *xp_get_cb(struct xdp_buff *xdp)
+{
+	return (void *)xdp + offsetof(struct xdp_buff_xsk, cb);
+}

This should have a wrapper in include/net/xdp_sock_drv.h that drivers will call.

Generally I think this should fly but I'm not sure about cb being 16 bytes.

Quoted 9 lines
include/net/xsk_buff_pool.h +1 -0
+
 void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb);
 static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb)

{

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

I'll send my patch to add support to mlx5 (using the drv_priv pointer approach) separately.

Saw them, thanks! Will include them in v3+.

#34 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

In reply to [email protected]
Quoted 30 lines · [email protected] writes:
Quoted 28 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 23 Nov 2022 10:26:41 -0800 Stanislav Fomichev wrote:
Quoted 3 lines

This embedding trick works for drivers that put xdp_buff on the stack, but mlx5 supports XSK zerocopy, which uses the xsk_buff_pool for allocating them. This makes it a bit awkward to do the same thing

there;

Quoted 3 lines
Quoted 3 lines

and since it's probably going to be fairly common to do something like this, how about we just add a 'void *drv_priv' pointer to struct xdp_buff that the drivers can use? The xdp_buff already takes up a

full

Quoted 1 line
Quoted 1 line

cache line anyway, so any data stuffed after it will spill over to a

new

Quoted 10 lines
Quoted 1 line

one; so I don't think there's much difference performance-wise.

I guess the alternative is to extend xsk_buff_pool with some new argument for xdp_buff tailroom? (so it can kmalloc(sizeof(xdp_buff) + xdp_buff_tailroom))
But it seems messy because there is no way of knowing what the target device's tailroom is, so it has to be a user setting :-/ I've started with a priv pointer in xdp_buff initially, it seems fine to go back. I'll probably convert veth/mlx4 to the same mode as well to avoid having different approaches in different places..

Can we not do this please? Add 16B of "private driver space" after the xdp_buff in xdp_buff_xsk (we have 16B to full cacheline), the drivers decide how they use it. Drivers can do BUILD_BUG_ON() for their expected size and cast that to whatever struct they want. This is how various offloads work, the variable size tailroom would be an over design IMO.

Quoted 8 lines
Quoted 1 line

And this way non XSK paths can keep its normal typing.

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

I am not suggesting to make anything variable-size; the 'void *drv_priv' is just a normal pointer. There's no changes to any typing; not sure where you got that from, Jakub?

Also, the priv pointer approach works for both XSK and on-stack allocations, unlike this approach (see below).

Quoted 46 lines
	dma_addr_t                 dma;                  /*    72     8 */
	dma_addr_t                 frame_dma;            /*    80     8 */
	struct xsk_buff_pool *     pool;                 /*    88     8 */
	u64                        orig_addr;            /*    96     8 */
	struct list_head           free_list_node;       /*   104    16 */
	/* size: 120, cachelines: 2, members: 7 */
	/* last cacheline: 56 bytes */
};

Toke, I can try to merge this into your patch + keep your SoB (or feel free to try this and retest yourself, whatever works).

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +5 -0
index bc2d9034af5b..837bf103b871 100644
@@ -44,6 +44,11 @@
  	(MLX5E_XDP_INLINE_WQE_MAX_DS_CNT * MLX5_SEND_WQE_DS - \
  	 sizeof(struct mlx5_wqe_inline_seg))
 
+struct mlx5_xdp_cb {
+	struct mlx5_cqe64 *cqe;
+	struct mlx5e_rq *rq;
+};
+
  struct mlx5e_xsk_param;
  int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param  
 *xsk);

bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct page *page,

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
index c91b54d9ff27..84d23b2da7ce 100644
@@ -5,6 +5,7 @@
  #include "en/xdp.h"
  #include <net/xdp_sock_drv.h>
  #include <linux/filter.h>
+#include <linux/build_bug.h>
 
  /* RX data path */
 
@@ -286,8 +287,14 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct  
 mlx5e_rq *rq,

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct · line 297

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +5 -0
  					      u32 cqe_bcnt)
  {
  	struct xdp_buff *xdp = wi->au->xsk;
+	struct mlx5_xdp_cb *cb;
  	struct bpf_prog *prog;
 
+	BUILD_BUG_ON(sizeof(struct mlx5_xdp_cb) > XSKB_CB_SIZE);
+	cb = xp_get_cb(xdp);
+	cb->cqe = NULL /*cqe*/;
+	cb->rq = rq;

So this works fine for the XSK path, but for the regular XDP path, mlx5 *does* indeed put the xdp_buff on the stack. So to re-use code there would be an implicit assumption that both memory layout and size matches between the two paths. I'm not sure that's better than just having a pointer inside the xdp_buff and pointing it wherever makes sense for that driver (as my patch did)?

-Toke

maintainer
#35 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Quoted 3 lines

I am not suggesting to make anything variable-size; the 'void *drv_priv' is just a normal pointer. There's no changes to any typing; not sure where you got that from, Jakub?

Often the descriptor pointer is in the same stack frame as the xdp_buff (or close enough). The point of adding the wrapping structure is to be able to move the descriptor pointer into a known place and then there's no extra store copying the descriptor pointer from one place on the stack to another.

#36 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

Quoted 3 lines

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Well, then you allocate a separate struct and point to that? Like I did in mlx5:

+ struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+ struct xdp_buff xdp = { .drv_priv = &mlctx };

but yeah, this does give an extra pointer deref on access. I'm not really opposed to the cb field either, I just think it's a bit odd to put it in struct xdp_buff_xsk; that basically requires the driver to keep the layouts in sync.

Instead, why not but a cb field into xdp_buff itself so it can be used for both the XSK and the non-XSK paths? Then the driver can just typecast the xdp_buff into its own struct that has whatever data it wants in place of the cb field?

-Toke

#37 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 32 lines · Toke Høiland-Jørgensen
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

Quoted 3 lines

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Well, then you allocate a separate struct and point to that? Like I did in mlx5:

(quoted diff) +2 -0
+	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+	struct xdp_buff xdp = { .drv_priv = &mlctx };

but yeah, this does give an extra pointer deref on access. I'm not really opposed to the cb field either, I just think it's a bit odd to put it in struct xdp_buff_xsk; that basically requires the driver to keep the layouts in sync.

Quoted 4 lines

Instead, why not but a cb field into xdp_buff itself so it can be used for both the XSK and the non-XSK paths? Then the driver can just typecast the xdp_buff into its own struct that has whatever data it wants in place of the cb field?

Why can't you simply have a pointer to xdp_buff in driver specific xdp_buff container which would point to xdp_buff that is stack based (or whatever else memory that will back it up - I am about to push a change that makes ice driver embed xdp_buff within a struct that represents Rx ring) for XDP path and for ZC the pointer to xdp_buff that you get from xsk_buff_pool ? This would satisfy both sides I believe and would let us keep the same container struct.

struct mlx4_xdp_buff {
	struct xdp_buff *xdp;
	struct mlx4_cqe *cqe;
	struct mlx4_en_dev *mdev;
	struct mlx4_en_rx_ring *ring;
	struct net_device *dev;
};

(...)

	struct mlx4_xdp_buff mxbuf;
	struct xdp_buff xdp;

	mxbuf.xdp = &xdp;
	xdp_init_buff(mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);

Also these additional things

+ mxbuf.cqe = cqe;
+ mxbuf.mdev = priv->mdev;
+ mxbuf.ring = ring;
+ mxbuf.dev = dev;

could be assigned once at a setup time or in worse case once per NAPI. So maybe mlx4_xdp_buff shouldn't be stack based?

#38 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 46 lines · Maciej Fijalkowski
Quoted 32 lines · Toke Høiland-Jørgensen
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

Quoted 3 lines

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Well, then you allocate a separate struct and point to that? Like I did in mlx5:

(quoted diff) +2 -0
+	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+	struct xdp_buff xdp = { .drv_priv = &mlctx };

but yeah, this does give an extra pointer deref on access. I'm not really opposed to the cb field either, I just think it's a bit odd to put it in struct xdp_buff_xsk; that basically requires the driver to keep the layouts in sync.

Quoted 4 lines

Instead, why not but a cb field into xdp_buff itself so it can be used for both the XSK and the non-XSK paths? Then the driver can just typecast the xdp_buff into its own struct that has whatever data it wants in place of the cb field?

Why can't you simply have a pointer to xdp_buff in driver specific xdp_buff container which would point to xdp_buff that is stack based (or whatever else memory that will back it up - I am about to push a change that makes ice driver embed xdp_buff within a struct that represents Rx ring) for XDP path and for ZC the pointer to xdp_buff that you get from xsk_buff_pool ? This would satisfy both sides I believe and would let us keep the same container struct.

Quoted 7 lines
struct mlx4_xdp_buff {
	struct xdp_buff *xdp;
	struct mlx4_cqe *cqe;
	struct mlx4_en_dev *mdev;
	struct mlx4_en_rx_ring *ring;
	struct net_device *dev;
};

Nah this won't work from kfunc POV, probably no way to retrieve the mlx4_xdp_buff based on xdp_buff ptr that needs to be used as an arg.

Sorry I'll think more about it, in the meantime let's hear more voices whether we should keep Stan's original approach + modify xdp_buff_xsk or go with Toke's proposal.

Quoted 18 lines

(...)

	struct mlx4_xdp_buff mxbuf;
	struct xdp_buff xdp;

	mxbuf.xdp = &xdp;
	xdp_init_buff(mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);

Also these additional things

(quoted diff) +4 -0
+			mxbuf.cqe = cqe;
+			mxbuf.mdev = priv->mdev;
+			mxbuf.ring = ring;
+			mxbuf.dev = dev;

could be assigned once at a setup time or in worse case once per NAPI. So maybe mlx4_xdp_buff shouldn't be stack based?

#39 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 55 lines · Maciej Fijalkowski
Quoted 46 lines · Maciej Fijalkowski
Quoted 32 lines · Toke Høiland-Jørgensen
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

	struct xdp_buff            xdp;                  /*     0    56 */
	u8                         cb[16];               /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

Quoted 3 lines

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Well, then you allocate a separate struct and point to that? Like I did in mlx5:

(quoted diff) +2 -0
+	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+	struct xdp_buff xdp = { .drv_priv = &mlctx };

but yeah, this does give an extra pointer deref on access. I'm not really opposed to the cb field either, I just think it's a bit odd to put it in struct xdp_buff_xsk; that basically requires the driver to keep the layouts in sync.

Quoted 4 lines

Instead, why not but a cb field into xdp_buff itself so it can be used for both the XSK and the non-XSK paths? Then the driver can just typecast the xdp_buff into its own struct that has whatever data it wants in place of the cb field?

Why can't you simply have a pointer to xdp_buff in driver specific xdp_buff container which would point to xdp_buff that is stack based (or whatever else memory that will back it up - I am about to push a change that makes ice driver embed xdp_buff within a struct that represents Rx ring) for XDP path and for ZC the pointer to xdp_buff that you get from xsk_buff_pool ? This would satisfy both sides I believe and would let us keep the same container struct.

Quoted 7 lines
struct mlx4_xdp_buff {
	struct xdp_buff *xdp;
	struct mlx4_cqe *cqe;
	struct mlx4_en_dev *mdev;
	struct mlx4_en_rx_ring *ring;
	struct net_device *dev;
};
Quoted 6 lines

Nah this won't work from kfunc POV, probably no way to retrieve the mlx4_xdp_buff based on xdp_buff ptr that needs to be used as an arg.

Sorry I'll think more about it, in the meantime let's hear more voices whether we should keep Stan's original approach + modify xdp_buff_xsk or go with Toke's proposal.

OK, so I played around with the mlx5 code a bit more, and I think the "wrapping struct + cb area" can be made to work without too many ugly casts; I'll send an updated version of the mlx5 patches with this incorporated tomorrow, after I've run some tests...

-Toke

#40 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 37 lines · Toke Høiland-Jørgensen
Quoted 34 lines · Maciej Fijalkowski
Quoted 33 lines · Maciej Fijalkowski
Quoted 32 lines · Toke Høiland-Jørgensen
Quoted 14 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
Quoted 6 lines

Good idea, prototyped below, lmk if it that's not what you had in mind.

struct xdp_buff_xsk {

      struct xdp_buff            xdp;                  /*     0    56 */
      u8                         cb[16];               /*    56    16 */
      /* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
Quoted 5 lines

As pahole helpfully says here, xdp_buff is actually only 8 bytes from being a full cache line. I thought about adding a 'cb' field like this to xdp_buff itself, but figured that since there's only room for a single pointer, why not just add that and let the driver point it to where it wants to store the extra context data?

Quoted 3 lines

What if the driver wants to store multiple pointers or an integer or whatever else? The single pointer seems quite arbitrary and not strictly necessary.

Well, then you allocate a separate struct and point to that? Like I did in mlx5:

(quoted diff) +2 -0
+  struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+  struct xdp_buff xdp = { .drv_priv = &mlctx };

but yeah, this does give an extra pointer deref on access. I'm not really opposed to the cb field either, I just think it's a bit odd to put it in struct xdp_buff_xsk; that basically requires the driver to keep the layouts in sync.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Instead, why not but a cb field into xdp_buff itself so it can be used for both the XSK and the non-XSK paths? Then the driver can just typecast the xdp_buff into its own struct that has whatever data it wants in place of the cb field?

Agreed, maybe having an explicit cb field in the xdp_buff is a nice compromise (assuming, over time, most devices will use it).

Quoted 19 lines
Quoted 19 lines
Quoted 8 lines

Why can't you simply have a pointer to xdp_buff in driver specific xdp_buff container which would point to xdp_buff that is stack based (or whatever else memory that will back it up - I am about to push a change that makes ice driver embed xdp_buff within a struct that represents Rx ring) for XDP path and for ZC the pointer to xdp_buff that you get from xsk_buff_pool ? This would satisfy both sides I believe and would let us keep the same container struct.

Quoted 7 lines
struct mlx4_xdp_buff {
     struct xdp_buff *xdp;
     struct mlx4_cqe *cqe;
     struct mlx4_en_dev *mdev;
     struct mlx4_en_rx_ring *ring;
     struct net_device *dev;
};

Nah this won't work from kfunc POV, probably no way to retrieve the mlx4_xdp_buff based on xdp_buff ptr that needs to be used as an arg.

Quoted 8 lines
Quoted 3 lines

Sorry I'll think more about it, in the meantime let's hear more voices whether we should keep Stan's original approach + modify xdp_buff_xsk or go with Toke's proposal.

OK, so I played around with the mlx5 code a bit more, and I think the "wrapping struct + cb area" can be made to work without too many ugly casts; I'll send an updated version of the mlx5 patches with this incorporated tomorrow, after I've run some tests...

I'll probably send a v3 sometime tomorrow (PST), so maybe wait for me to make sure we are working on the same base?
Or LMK if you prefer to do it differently..

#41 lore raw

Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

Quoted 23 lines · Stanislav Fomichev
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

Why can't you simply have a pointer to xdp_buff in driver specific xdp_buff container which would point to xdp_buff that is stack based (or whatever else memory that will back it up - I am about to push a change that makes ice driver embed xdp_buff within a struct that represents Rx ring) for XDP path and for ZC the pointer to xdp_buff that you get from xsk_buff_pool ? This would satisfy both sides I believe and would let us keep the same container struct.

Quoted 7 lines
struct mlx4_xdp_buff {
     struct xdp_buff *xdp;
     struct mlx4_cqe *cqe;
     struct mlx4_en_dev *mdev;
     struct mlx4_en_rx_ring *ring;
     struct net_device *dev;
};

Nah this won't work from kfunc POV, probably no way to retrieve the mlx4_xdp_buff based on xdp_buff ptr that needs to be used as an arg.

Sorry I'll think more about it, in the meantime let's hear more voices whether we should keep Stan's original approach + modify xdp_buff_xsk or go with Toke's proposal.

Quoted 8 lines
Quoted 4 lines

OK, so I played around with the mlx5 code a bit more, and I think the "wrapping struct + cb area" can be made to work without too many ugly casts; I'll send an updated version of the mlx5 patches with this incorporated tomorrow, after I've run some tests...

I'll probably send a v3 sometime tomorrow (PST), so maybe wait for me to make sure we are working on the same base?
Or LMK if you prefer to do it differently..

OK, I'll send you my mlx5 patches off-list so you can just incorporate those. Got stuck on some annoying build issues for the perf testing, so will defer that until your next version, then :)

-Toke

#42 lore raw

PATCH v2 7/8 mxl4: Support RX XDP metadata

In reply to Stanislav Fomichev

RX timestamp and hash for now. Tested using the prog from the next patch.

Also enabling xdp metadata support; don't see why it's disabled, there is enough headroom..

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 10 ++++
 drivers/net/ethernet/mellanox/mlx4/en_rx.c    | 48 ++++++++++++++++++-
 include/linux/mlx4/device.h                   |  7 +++
 3 files changed, 64 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +10 -0
index 8800d3f1f55c..1cb63746a851 100644
@@ -2855,6 +2855,11 @@ static const struct net_device_ops mlx4_netdev_ops = {
 	.ndo_features_check	= mlx4_en_features_check,
 	.ndo_set_tx_maxrate	= mlx4_en_set_tx_maxrate,
 	.ndo_bpf		= mlx4_xdp,
+
+	.ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp	= mlx4_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash	= mlx4_xdp_rx_hash,
 };
 
 static const struct net_device_ops mlx4_netdev_ops_master = {
@@ -2887,6 +2892,11 @@ static const struct net_device_ops mlx4_netdev_ops_master = {
 	.ndo_features_check	= mlx4_en_features_check,
 	.ndo_set_tx_maxrate	= mlx4_en_set_tx_maxrate,
 	.ndo_bpf		= mlx4_xdp,
+
+	.ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp	= mlx4_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash	= mlx4_xdp_rx_hash,
 };
 
 struct mlx4_en_bond {
drivers/net/ethernet/mellanox/mlx4/en_rx.c +47 -1

63 lines not on this page · raw message

include/linux/mlx4/device.h +7 -0
index 6646634a0b9d..d5904da1d490 100644
@@ -1585,4 +1585,11 @@ static inline int mlx4_get_num_reserved_uar(struct mlx4_dev *dev)
 	/* The first 128 UARs are used for EQ doorbells */
 	return (128 >> (PAGE_SHIFT - dev->uar_page_shift));
 }
+
+struct xdp_md;
+bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx);
+u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx);
+bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx);
+u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx);
+
 #endif /* MLX4_DEVICE_H */
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#43 lore raw

Re: [PATCH bpf-next v2 7/8] mxl4: Support RX XDP metadata

Quoted 79 lines · Stanislav Fomichev

RX timestamp and hash for now. Tested using the prog from the next patch.

Also enabling xdp metadata support; don't see why it's disabled, there is enough headroom..

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
  .../net/ethernet/mellanox/mlx4/en_netdev.c    | 10 ++++
  drivers/net/ethernet/mellanox/mlx4/en_rx.c    | 48 ++++++++++++++++++-
  include/linux/mlx4/device.h                   |  7 +++
  3 files changed, 64 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +10 -0
index 8800d3f1f55c..1cb63746a851 100644
@@ -2855,6 +2855,11 @@ static const struct net_device_ops mlx4_netdev_ops = {
  	.ndo_features_check	= mlx4_en_features_check,
  	.ndo_set_tx_maxrate	= mlx4_en_set_tx_maxrate,
  	.ndo_bpf		= mlx4_xdp,
+
+	.ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp	= mlx4_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash	= mlx4_xdp_rx_hash,
  };
 
  static const struct net_device_ops mlx4_netdev_ops_master = {
@@ -2887,6 +2892,11 @@ static const struct net_device_ops mlx4_netdev_ops_master = {
  	.ndo_features_check	= mlx4_en_features_check,
  	.ndo_set_tx_maxrate	= mlx4_en_set_tx_maxrate,
  	.ndo_bpf		= mlx4_xdp,
+
+	.ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp	= mlx4_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash	= mlx4_xdp_rx_hash,
  };
 
  struct mlx4_en_bond {
drivers/net/ethernet/mellanox/mlx4/en_rx.c +15 -0
index 467356633172..fd14d59f6cbf 100644
@@ -663,8 +663,50 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
 
  struct mlx4_xdp_buff {
  	struct xdp_buff xdp;
+	struct mlx4_cqe *cqe;
+	struct mlx4_en_dev *mdev;
+	struct mlx4_en_rx_ring *ring;
+	struct net_device *dev;
  };
 
+bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx)
+{
+	struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+	return _ctx->ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL;
+}
+
+u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx)
+{
+	struct mlx4_xdp_buff *_ctx = (void *)ctx;
+	unsigned int seq;

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, · line 692

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +10 -0
+	u64 timestamp;
+	u64 nsec;
+
+	timestamp = mlx4_en_get_cqe_ts(_ctx->cqe);
+
+	do {
+		seq = read_seqbegin(&_ctx->mdev->clock_lock);
+		nsec = timecounter_cyc2time(&_ctx->mdev->clock, timestamp);
+	} while (read_seqretry(&_ctx->mdev->clock_lock, seq));
+

This is open-code version of mlx4_en_fill_hwtstamps. Better use the existing function.

Quoted 50 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +22 -1
+	return ns_to_ktime(nsec);
+}
+
+bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx)
+{
+	struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+	return _ctx->dev->features & NETIF_F_RXHASH;
+}
+
+u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx)
+{
+	struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+	return be32_to_cpu(_ctx->cqe->immed_rss_invalid);
+}
+
  int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)
  {
  	struct mlx4_en_priv *priv = netdev_priv(dev);
@@ -781,8 +823,12 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
  						DMA_FROM_DEVICE);
 
  			xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,
-					 frags[0].page_offset, length, false);
+					 frags[0].page_offset, length, true);
  			orig_data = mxbuf.xdp.data;
+			mxbuf.cqe = cqe;
+			mxbuf.mdev = priv->mdev;
+			mxbuf.ring = ring;
+			mxbuf.dev = dev;
 
  			act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);
 
include/linux/mlx4/device.h +7 -0
index 6646634a0b9d..d5904da1d490 100644
@@ -1585,4 +1585,11 @@ static inline int mlx4_get_num_reserved_uar(struct mlx4_dev *dev)
  	/* The first 128 UARs are used for EQ doorbells */
  	return (128 >> (PAGE_SHIFT - dev->uar_page_shift));
  }
+
+struct xdp_md;
+bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx);
+u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx);
+bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx);
+u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx);
+
  #endif /* MLX4_DEVICE_H */
#44 lore raw

Re: [PATCH bpf-next v2 7/8] mxl4: Support RX XDP metadata

Quoted 88 lines · Tariq Toukan
Quoted 84 lines · Stanislav Fomichev

RX timestamp and hash for now. Tested using the prog from the next patch.

Also enabling xdp metadata support; don't see why it's disabled, there is enough headroom..

Cc Tariq Toukan <[email protected]>
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
  .../net/ethernet/mellanox/mlx4/en_netdev.c    | 10 ++++
  drivers/net/ethernet/mellanox/mlx4/en_rx.c    | 48 ++++++++++++++++++-
  include/linux/mlx4/device.h                   |  7 +++
  3 files changed, 64 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +10 -0
index 8800d3f1f55c..1cb63746a851 100644
@@ -2855,6 +2855,11 @@ static const struct net_device_ops mlx4_netdev_ops = {
      .ndo_features_check     = mlx4_en_features_check,
      .ndo_set_tx_maxrate     = mlx4_en_set_tx_maxrate,
      .ndo_bpf                = mlx4_xdp,
+
+     .ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+     .ndo_xdp_rx_timestamp   = mlx4_xdp_rx_timestamp,
+     .ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+     .ndo_xdp_rx_hash        = mlx4_xdp_rx_hash,
  };
 
  static const struct net_device_ops mlx4_netdev_ops_master = {
@@ -2887,6 +2892,11 @@ static const struct net_device_ops mlx4_netdev_ops_master = {
      .ndo_features_check     = mlx4_en_features_check,
      .ndo_set_tx_maxrate     = mlx4_en_set_tx_maxrate,
      .ndo_bpf                = mlx4_xdp,
+
+     .ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported,
+     .ndo_xdp_rx_timestamp   = mlx4_xdp_rx_timestamp,
+     .ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported,
+     .ndo_xdp_rx_hash        = mlx4_xdp_rx_hash,
  };
 
  struct mlx4_en_bond {
drivers/net/ethernet/mellanox/mlx4/en_rx.c +20 -0
index 467356633172..fd14d59f6cbf 100644
@@ -663,8 +663,50 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,
 
  struct mlx4_xdp_buff {
      struct xdp_buff xdp;
+     struct mlx4_cqe *cqe;
+     struct mlx4_en_dev *mdev;
+     struct mlx4_en_rx_ring *ring;
+     struct net_device *dev;
  };
 
+bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx)
+{
+     struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+     return _ctx->ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL;
+}
+
+u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx)
+{
+     struct mlx4_xdp_buff *_ctx = (void *)ctx;
+     unsigned int seq;
+     u64 timestamp;
+     u64 nsec;
+
+     timestamp = mlx4_en_get_cqe_ts(_ctx->cqe);
+
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx4/en_rx.c static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, · line 692

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +5 -0
+     do {
+             seq = read_seqbegin(&_ctx->mdev->clock_lock);
+             nsec = timecounter_cyc2time(&_ctx->mdev->clock, timestamp);
+     } while (read_seqretry(&_ctx->mdev->clock_lock, seq));
+

This is open-code version of mlx4_en_fill_hwtstamps. Better use the existing function.

That one assumes the skb_shared_hwtstamps argument :-(
Should I try to separate the common parts into some new helper function instead?

Or maybe I can just change mlx4_en_fill_hwtstamps to the following?

u64 mlx4_en_fill_hwtstamps(struct mlx4_en_dev *mdev, u64 timestamp) {
...
return ns_to_ktime(nsec);
}

And replace existing callers with:

skb_hwtstamps(skb)->hwtstamp = mlx4_en_fill_hwtstamps(priv->mdev, timestamp).

?

Quoted 50 lines
Quoted 50 lines
drivers/net/ethernet/mellanox/mlx4/en_rx.c +22 -1
+     return ns_to_ktime(nsec);
+}
+
+bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx)
+{
+     struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+     return _ctx->dev->features & NETIF_F_RXHASH;
+}
+
+u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx)
+{
+     struct mlx4_xdp_buff *_ctx = (void *)ctx;
+
+     return be32_to_cpu(_ctx->cqe->immed_rss_invalid);
+}
+
  int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)
  {
      struct mlx4_en_priv *priv = netdev_priv(dev);
@@ -781,8 +823,12 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
                                              DMA_FROM_DEVICE);
 
                      xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,
-                                      frags[0].page_offset, length, false);
+                                      frags[0].page_offset, length, true);
                      orig_data = mxbuf.xdp.data;
+                     mxbuf.cqe = cqe;
+                     mxbuf.mdev = priv->mdev;
+                     mxbuf.ring = ring;
+                     mxbuf.dev = dev;
 
                      act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);
 
include/linux/mlx4/device.h +7 -0
index 6646634a0b9d..d5904da1d490 100644
@@ -1585,4 +1585,11 @@ static inline int mlx4_get_num_reserved_uar(struct mlx4_dev *dev)
      /* The first 128 UARs are used for EQ doorbells */
      return (128 >> (PAGE_SHIFT - dev->uar_page_shift));
  }
+
+struct xdp_md;
+bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx);
+u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx);
+bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx);
+u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx);
+
  #endif /* MLX4_DEVICE_H */
#45 lore raw

PATCH v2 8/8 selftests/bpf: Simple program to dump XDP RX metadata

In reply to Stanislav Fomichev

To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.

$ xdp_hw_metadata <ifname>

On the other machine:

$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDP
$ echo -n skb | nc -u -q1 <target> 9092 # for skb

Sample output:

  # xdp
  xsk_ring_cons__peek: 1
  0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000
  rx_timestamp_supported: 1
  rx_timestamp: 1667850075063948829
  0x19f9090: complete idx=8 addr=8000
  # skb
  found skb hwtstamp = 1668314052.854274681

Decoding:

  # xdp
  rx_timestamp=1667850075.063948829
  $ date -d @1667850075
  Mon Nov  7 11:41:15 AM PST 2022
  $ date
  Mon Nov  7 11:42:05 AM PST 2022
  # skb
  $ date -d @1668314052
  Sat Nov 12 08:34:12 PM PST 2022
  $ date
  Sat Nov 12 08:37:06 PM PST 2022
Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
---
 tools/testing/selftests/bpf/.gitignore        |   1 +
 tools/testing/selftests/bpf/Makefile          |   6 +-
 .../selftests/bpf/progs/xdp_hw_metadata.c     |  93 ++++
 tools/testing/selftests/bpf/xdp_hw_metadata.c | 405 ++++++++++++++++++
 4 files changed, 504 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/xdp_hw_metadata.c
 create mode 100644 tools/testing/selftests/bpf/xdp_hw_metadata.c
tools/testing/selftests/bpf/.gitignore +1 -0
index 07d2d0a8c5cb..01e3baeefd4f 100644
@@ -46,3 +46,4 @@ test_cpp
 xskxceiver
 xdp_redirect_multi
 xdp_synproxy
+xdp_hw_metadata
tools/testing/selftests/bpf/Makefile +5 -1
index 4eed22fa3681..189b39b0e5d0 100644
@@ -83,7 +83,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \
 TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \
 	flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \
 	test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \
-	xskxceiver xdp_redirect_multi xdp_synproxy veristat
+	xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata
 
 TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read $(OUTPUT)/sign-file
 TEST_GEN_FILES += liburandom_read.so
@@ -241,6 +241,9 @@ $(OUTPUT)/test_maps: $(TESTING_HELPERS)
 $(OUTPUT)/test_verifier: $(TESTING_HELPERS) $(CAP_HELPERS)
 $(OUTPUT)/xsk.o: $(BPFOBJ)
 $(OUTPUT)/xskxceiver: $(OUTPUT)/xsk.o
+$(OUTPUT)/xdp_hw_metadata: $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h
+$(OUTPUT)/xdp_hw_metadata: $(OUTPUT)/network_helpers.o
+$(OUTPUT)/xdp_hw_metadata: LDFLAGS += -static
 
 BPFTOOL ?= $(DEFAULT_BPFTOOL)
 $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile)    \
@@ -383,6 +386,7 @@ linked_maps.skel.h-deps := linked_maps1.bpf.o linked_maps2.bpf.o
 test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_subskeleton.bpf.o
 test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o
 test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o
+xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o
 
 LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
 
tools/testing/selftests/bpf/progs/xdp_hw_metadata.c new file +93 -0

93 lines not on this page · raw message

tools/testing/selftests/bpf/xdp_hw_metadata.c new file +405 -0

405 lines not on this page · raw message

Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#46 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 8/8] selftests/bpf: Simple program to dump XDP RX metadata

Quoted 14 lines · Stanislav Fomichev
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +14 -0
+static int rxq_num(const char *ifname)
+{
+	struct ethtool_channels ch = {
+		.cmd = ETHTOOL_GCHANNELS,
+	};
+
+	struct ifreq ifr = {
+		.ifr_data = (void *)&ch,
+	};
+	strcpy(ifr.ifr_name, ifname);
+	int fd, ret;
+
+	fd = socket(AF_UNIX, SOCK_DGRAM, 0);
+	if (fd < 0)

tools/testing/selftests/bpf/xdp_hw_metadata.c line 288

Quoted 10 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +10 -0
+		error(-1, errno, "socket");
+
+	ret = ioctl(fd, SIOCETHTOOL, &ifr);
+	if (ret < 0)
+		error(-1, errno, "socket");
+
+	close(fd);
+
+	return ch.rx_count;
+}

mlx5 uses 'combined' channels, so this returns 0. Changing it to just:

return ch.rx_count ?: ch.combined_count;

works though :)

-Toke

#47 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 8/8] selftests/bpf: Simple program to dump XDP RX metadata

Quoted 30 lines · Toke Høiland-Jørgensen
Quoted 14 lines · Stanislav Fomichev
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +14 -0
+static int rxq_num(const char *ifname)
+{
+     struct ethtool_channels ch = {
+             .cmd = ETHTOOL_GCHANNELS,
+     };
+
+     struct ifreq ifr = {
+             .ifr_data = (void *)&ch,
+     };
+     strcpy(ifr.ifr_name, ifname);
+     int fd, ret;
+
+     fd = socket(AF_UNIX, SOCK_DGRAM, 0);
+     if (fd < 0)

tools/testing/selftests/bpf/xdp_hw_metadata.c line 288

Quoted 10 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +10 -0
+             error(-1, errno, "socket");
+
+     ret = ioctl(fd, SIOCETHTOOL, &ifr);
+     if (ret < 0)
+             error(-1, errno, "socket");
+
+     close(fd);
+
+     return ch.rx_count;
+}

mlx5 uses 'combined' channels, so this returns 0. Changing it to just:

Quoted 3 lines

return ch.rx_count ?: ch.combined_count;

works though :)

Perfect, will do the same :-) Thank you for running and testing!

Quoted 2 lines

-Toke

maintainer
#48 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 8/8] selftests/bpf: Simple program to dump XDP RX metadata

Quoted 5 lines · On Wed, 23 Nov 2022 10:29:23 -0800 Stanislav Fomichev wrote:
Quoted 3 lines

return ch.rx_count ?: ch.combined_count;

works though :)

Perfect, will do the same :-) Thank you for running and testing!

The correct value is ch.rx_count + ch.combined_count

We've been over this many times, I thought it was coded up in libbpf but I don't see it now :S

#49 lore raw

Re: [xdp-hints] [PATCH bpf-next v2 8/8] selftests/bpf: Simple program to dump XDP RX metadata

Quoted 4 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 23 Nov 2022 10:29:23 -0800 Stanislav Fomichev wrote:
Quoted 2 lines

return ch.rx_count ?: ch.combined_count;

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

works though :)

Perfect, will do the same :-) Thank you for running and testing!

The correct value is ch.rx_count + ch.combined_count

We've been over this many times, I thought it was coded up in libbpf but I don't see it now :S

Yeah, can't find it. Also doesn't exist on libxdp/libxsk. Will apply your fix, thank you!

#50 lore raw

PATCH 1/2 xdp: Add drv_priv pointer to struct xdp_buff

In reply to Stanislav Fomichev

This allows drivers to add more context data to the xdp_buff object, which they can use for metadata kfunc implementations.

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
Cc Stanislav Fomichev <[email protected]>
---
 include/net/xdp.h | 1 +
 1 file changed, 1 insertion(+)
include/net/xdp.h +1 -0
index 348aefd467ed..27c54ad3c8e2 100644
@@ -84,6 +84,7 @@ struct xdp_buff {
 	struct xdp_txq_info *txq;
 	u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/
 	u32 flags; /* supported values defined in xdp_buff_flags */
+	void *drv_priv;
 };
 
 static __always_inline bool xdp_buff_has_frags(struct xdp_buff *xdp)
Signature · 1 line
2.38.1
#51 lore raw

PATCH 2/2 mlx5: Support XDP RX metadata

Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
Cc Stanislav Fomichev <[email protected]>

---
This goes on top of Stanislav's series, obvioulsy. Verified that it works using the xdp_hw_metadata utility; going to do ome benchmarking and follow up with the results, but figured I'd send this out straight away in case others wanted to play with it.

Stanislav, feel free to fold it into the next version of your series if you want!

-Toke

 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  7 +++-
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 32 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  | 10 ++++++
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |  3 ++
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |  3 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  4 +++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 19 +++++------
 7 files changed, 65 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +6 -1
index ff5b302531d5..960404027f0b 100644
@@ -629,7 +629,7 @@ typedef struct sk_buff *
 			       u16 cqe_bcnt, u32 head_offset, u32 page_idx);
 typedef struct sk_buff *
 (*mlx5e_fp_skb_from_cqe)(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,
-			 u32 cqe_bcnt);
+			 struct mlx5_cqe64 *cqe, u32 cqe_bcnt);
 typedef bool (*mlx5e_fp_post_rx_wqes)(struct mlx5e_rq *rq);
 typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq*, u16);
 typedef void (*mlx5e_fp_shampo_dealloc_hd)(struct mlx5e_rq*, u16, u16, bool);
@@ -1035,6 +1035,11 @@ int mlx5e_vlan_rx_kill_vid(struct net_device *dev, __always_unused __be16 proto,
 			   u16 vid);
 void mlx5e_timestamp_init(struct mlx5e_priv *priv);
 
+static inline bool mlx5e_rx_hw_stamp(struct hwtstamp_config *config)
+{
+	return config->rx_filter == HWTSTAMP_FILTER_ALL;
+}
+
 struct mlx5e_xsk_param;
 
 struct mlx5e_rq_param;
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +32 -0
index 20507ef2f956..604c8cdfde02 100644
@@ -156,6 +156,38 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
 	return true;
 }
 
+bool mlx5e_xdp_rx_timestamp_supported(const struct xdp_md *ctx)
+{
+	const struct xdp_buff *xdp = (void *)ctx;
+	struct mlx5_xdp_ctx *mctx = xdp->drv_priv;
+
+	return mlx5e_rx_hw_stamp(mctx->rq->tstamp);
+}
+
+u64 mlx5e_xdp_rx_timestamp(const struct xdp_md *ctx)
+{
+	const struct xdp_buff *xdp = (void *)ctx;
+	struct mlx5_xdp_ctx *mctx = xdp->drv_priv;
+
+	return mlx5e_cqe_ts_to_ns(mctx->rq->ptp_cyc2time,
+				  mctx->rq->clock, get_cqe_ts(mctx->cqe));
+}
+
+bool mlx5e_xdp_rx_hash_supported(const struct xdp_md *ctx)
+{
+	const struct xdp_buff *xdp = (void *)ctx;
+
+	return xdp->rxq->dev->features & NETIF_F_RXHASH;
+}
+
+u32 mlx5e_xdp_rx_hash(const struct xdp_md *ctx)
+{
+	const struct xdp_buff *xdp = (void *)ctx;
+	struct mlx5_xdp_ctx *mctx = xdp->drv_priv;
+
+	return be32_to_cpu(mctx->cqe->rss_hash_result);
+}
+
 /* returns true if packet was consumed by xdp */
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct page *page,
 		      struct bpf_prog *prog, struct xdp_buff *xdp)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +10 -0
index bc2d9034af5b..07d80d0446ff 100644
@@ -44,6 +44,11 @@
 	(MLX5E_XDP_INLINE_WQE_MAX_DS_CNT * MLX5_SEND_WQE_DS - \
 	 sizeof(struct mlx5_wqe_inline_seg))
 
+struct mlx5_xdp_ctx {
+	struct mlx5_cqe64 *cqe;
+	struct mlx5e_rq *rq;
+};
+
 struct mlx5e_xsk_param;
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk);
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct page *page,
@@ -56,6 +61,11 @@ void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq);
 int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
 		   u32 flags);
 
+bool mlx5e_xdp_rx_hash_supported(const struct xdp_md *ctx);
+u32 mlx5e_xdp_rx_hash(const struct xdp_md *ctx);
+bool mlx5e_xdp_rx_timestamp_supported(const struct xdp_md *ctx);
+u64 mlx5e_xdp_rx_timestamp(const struct xdp_md *ctx);
+
 INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 							  struct mlx5e_xmit_data *xdptxd,
 							  struct skb_shared_info *sinfo,
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +3 -0
index c91b54d9ff27..c6715cb23d45 100644
@@ -283,8 +283,10 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 
 struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
 					      struct mlx5e_wqe_frag_info *wi,
+					      struct mlx5_cqe64 *cqe,
 					      u32 cqe_bcnt)
 {
+	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
 	struct xdp_buff *xdp = wi->au->xsk;
 	struct bpf_prog *prog;
 
@@ -298,6 +300,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
 	xsk_buff_set_size(xdp, cqe_bcnt);
 	xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool);
 	net_prefetch(xdp->data);
+	xdp->drv_priv = &mlctx;
 
 	prog = rcu_dereference(rq->xdp_prog);
 	if (likely(prog && mlx5e_xdp_handle(rq, NULL, prog, xdp)))
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +2 -1
index 087c943bd8e9..9198f137f48f 100644
@@ -18,6 +18,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 						    u32 page_idx);
 struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
 					      struct mlx5e_wqe_frag_info *wi,
-					      u32 cqe_bcnt);
+                                              struct mlx5_cqe64 *cqe,
+                                              u32 cqe_bcnt);
 
 #endif /* __MLX5_EN_XSK_RX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
index 14bd86e368d5..015bfe891458 100644
@@ -4890,6 +4890,10 @@ const struct net_device_ops mlx5e_netdev_ops = {
 	.ndo_tx_timeout          = mlx5e_tx_timeout,
 	.ndo_bpf		 = mlx5e_xdp,
 	.ndo_xdp_xmit            = mlx5e_xdp_xmit,
+	.ndo_xdp_rx_timestamp_supported = mlx5e_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp    = mlx5e_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx5e_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash         = mlx5e_xdp_rx_hash,
 	.ndo_xsk_wakeup          = mlx5e_xsk_wakeup,
 #ifdef CONFIG_MLX5_EN_ARFS
 	.ndo_rx_flow_steer	 = mlx5e_rx_flow_steer,
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +8 -11
index b1ea0b995d9c..1d6600441e74 100644
@@ -76,11 +76,6 @@ const struct mlx5e_rx_handlers mlx5e_rx_handlers_nic = {
 	.handle_rx_cqe_mpwqe_shampo = mlx5e_handle_rx_cqe_mpwrq_shampo,
 };
 
-static inline bool mlx5e_rx_hw_stamp(struct hwtstamp_config *config)
-{
-	return config->rx_filter == HWTSTAMP_FILTER_ALL;
-}
-
 static inline void mlx5e_read_cqe_slot(struct mlx5_cqwq *wq,
 				       u32 cqcc, void *data)
 {
@@ -1573,7 +1568,7 @@ static void mlx5e_fill_xdp_buff(struct mlx5e_rq *rq, void *va, u16 headroom,
 
 static struct sk_buff *
 mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,
-			  u32 cqe_bcnt)
+			  struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
 {
 	union mlx5e_alloc_unit *au = wi->au;
 	u16 rx_headroom = rq->buff.headroom;
@@ -1595,7 +1590,8 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,
 
 	prog = rcu_dereference(rq->xdp_prog);
 	if (prog) {
-		struct xdp_buff xdp;
+		struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+		struct xdp_buff xdp = { .drv_priv = &mlctx };
 
 		net_prefetchw(va); /* xdp_frame data area */
 		mlx5e_fill_xdp_buff(rq, va, rx_headroom, cqe_bcnt, &xdp);
@@ -1619,16 +1615,17 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,
 
 static struct sk_buff *
 mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,
-			     u32 cqe_bcnt)
+			     struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
 {
 	struct mlx5e_rq_frag_info *frag_info = &rq->wqe.info.arr[0];
+	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
+	struct xdp_buff xdp = { .drv_priv = &mlctx };
 	struct mlx5e_wqe_frag_info *head_wi = wi;
 	union mlx5e_alloc_unit *au = wi->au;
 	u16 rx_headroom = rq->buff.headroom;
 	struct skb_shared_info *sinfo;
 	u32 frag_consumed_bytes;
 	struct bpf_prog *prog;
-	struct xdp_buff xdp;
 	struct sk_buff *skb;
 	dma_addr_t addr;
 	u32 truesize;
@@ -1766,7 +1763,7 @@ static void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 			      mlx5e_skb_from_cqe_linear,
 			      mlx5e_skb_from_cqe_nonlinear,
 			      mlx5e_xsk_skb_from_cqe_linear,
-			      rq, wi, cqe_bcnt);
+			      rq, wi, cqe, cqe_bcnt);
 	if (!skb) {
 		/* probably for XDP */
 		if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) {
@@ -2575,7 +2572,7 @@ static void mlx5e_trap_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe
 		goto free_wqe;
 	}
 
-	skb = mlx5e_skb_from_cqe_nonlinear(rq, wi, cqe_bcnt);
+	skb = mlx5e_skb_from_cqe_nonlinear(rq, wi, cqe, cqe_bcnt);
 	if (!skb)
 		goto free_wqe;
 
Signature · 1 line
2.38.1
#52 lore raw

Re: [xdp-hints] [PATCH bpf-next 2/2] mlx5: Support XDP RX metadata

Quoted 27 lines · Toke Høiland-Jørgensen

Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
Cc Stanislav Fomichev <[email protected]>

---
This goes on top of Stanislav's series, obvioulsy. Verified that it works using the xdp_hw_metadata utility; going to do ome benchmarking and follow up with the results, but figured I'd send this out straight away in case others wanted to play with it.

Stanislav, feel free to fold it into the next version of your series if you want!

[...]

drivers/net/ethernet/mellanox/mlx5/core/en_main.c const struct net_device_ops mlx5e_netdev_ops = { · line 4896

Quoted 13 lines

#endif /* __MLX5_EN_XSK_RX_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
index 14bd86e368d5..015bfe891458 100644
@@ -4890,6 +4890,10 @@ const struct net_device_ops mlx5e_netdev_ops = {
 	.ndo_tx_timeout          = mlx5e_tx_timeout,
 	.ndo_bpf		 = mlx5e_xdp,
 	.ndo_xdp_xmit            = mlx5e_xdp_xmit,
+	.ndo_xdp_rx_timestamp_supported = mlx5e_xdp_rx_timestamp_supported,
+	.ndo_xdp_rx_timestamp    = mlx5e_xdp_rx_timestamp,
+	.ndo_xdp_rx_hash_supported = mlx5e_xdp_rx_hash_supported,
+	.ndo_xdp_rx_hash         = mlx5e_xdp_rx_hash,

I hope i am not late to the party.
but I already expressed my feelings regarding using kfunc for xdp hints, @LPC and @netdevconf.

I think it's wrong to use indirect calls, and for many usecases the overhead will be higher than just calculating the metadata on the spot.

so you will need two indirect calls per packet per hint.. some would argue on some systems calculating the hash would be much faster. and one major reason to have the hints is to accelerate xdp edge and
security programs with the hw provided hints.

what happened with just asking the driver to place the data in a specific location on the headroom?

#53 lore raw

Re: [xdp-hints] [PATCH bpf-next 2/2] mlx5: Support XDP RX metadata

Quoted 53 lines · Saeed Mahameed
Quoted 27 lines · Toke Høiland-Jørgensen

Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.

Cc John Fastabend <[email protected]>
Cc David Ahern <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Jesper Dangaard Brouer <[email protected]>
Cc Anatoly Burakov <[email protected]>
Cc Alexander Lobakin <[email protected]>
Cc Magnus Karlsson <[email protected]>
Cc Maryam Tahhan <[email protected]>
Cc Stanislav Fomichev <[email protected]>

---
This goes on top of Stanislav's series, obvioulsy. Verified that it works using the xdp_hw_metadata utility; going to do ome benchmarking and follow up with the results, but figured I'd send this out straight away in case others wanted to play with it.

Stanislav, feel free to fold it into the next version of your series if you want!

[...]

drivers/net/ethernet/mellanox/mlx5/core/en_main.c const struct net_device_ops mlx5e_netdev_ops = { · line 4896

Quoted 13 lines

#endif /* __MLX5_EN_XSK_RX_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
index 14bd86e368d5..015bfe891458 100644
@@ -4890,6 +4890,10 @@ const struct net_device_ops mlx5e_netdev_ops = {
      .ndo_tx_timeout          = mlx5e_tx_timeout,
      .ndo_bpf                 = mlx5e_xdp,
      .ndo_xdp_xmit            = mlx5e_xdp_xmit,
+      .ndo_xdp_rx_timestamp_supported = mlx5e_xdp_rx_timestamp_supported,
+      .ndo_xdp_rx_timestamp    = mlx5e_xdp_rx_timestamp,
+      .ndo_xdp_rx_hash_supported = mlx5e_xdp_rx_hash_supported,
+      .ndo_xdp_rx_hash         = mlx5e_xdp_rx_hash,

I hope i am not late to the party.
but I already expressed my feelings regarding using kfunc for xdp hints, @LPC and @netdevconf.

I think it's wrong to use indirect calls, and for many usecases the overhead will be higher than just calculating the metadata on the spot.

Quoted 7 lines

so you will need two indirect calls per packet per hint..
some would argue on some systems calculating the hash would be much faster. and one major reason to have the hints is to accelerate xdp edge and security programs with the hw provided hints.

what happened with just asking the driver to place the data in a specific location on the headroom?

Take a look at [0], we are resolving indirect calls. We can also always go back to unrolling those calls as was done initially in [1].

0: https://lore.kernel.org/bpf/[email protected]/
1: https://lore.kernel.org/bpf/[email protected]/

kfunc approach seems more flexible than an all-or-nothing approach with the driver pre-filling all metadata.