Skip to contentSkip to conversation

net/mlx5: use indirect call wrappers

PATCH Applied Paolo Abeni 3 participants 6 messages Jun 12, 2019 – Jun 14, 2019

Subsystems (4)
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

v3 cover net/mlx5: use indirect call wrappers

The mlx5_core driver uses several indirect calls in fast-path, some of them are invoked on each ingress packet, even for the XDP-only traffic.

This series leverage the indirect call wrappers infrastructure the avoid the expansive RETPOLINE overhead for 2 indirect calls in fast-path.

Each call is addressed on a different patch, plus we need to introduce a couple of additional helpers to cope with the higher number of possible direct-call alternatives.

v2 -> v3:
- do not add more INDIRECT_CALL_* macros
- use only the direct calls always available regardless of the mlx5 build options in the last patch

v1 -> v2:
- update the direct call list and use a macro to define it,
as per Saeed suggestion. An intermediated additional
macro is needed to allow arg list expansion
- patch 2/3 is unchanged, as the generated code looks better this way than with possible alternative (dropping BP hits)

Paolo Abeni (2):
net/mlx5e: use indirect calls wrapper for skb allocation net/mlx5e: use indirect calls wrapper for the rx packet handler

 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 25 ++++++++++++++-----
 1 file changed, 19 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
author maintainer
#1 lore raw

PATCH v3 1/2 net/mlx5e: use indirect calls wrapper for skb allocation

We can avoid an indirect call per packet wrapping the skb creation with the appropriate helper.

---
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 22 ++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +17 -5
index 13133e7f088e..0fe5f13d07cc 100644
@@ -34,6 +34,7 @@
 #include <linux/ip.h>
 #include <linux/ipv6.h>
 #include <linux/tcp.h>
+#include <linux/indirect_call_wrapper.h>
 #include <net/ip6_checksum.h>
 #include <net/page_pool.h>
 #include <net/inet_ecn.h>
@@ -1092,7 +1093,10 @@ void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 	wi       = get_frag(rq, ci);
 	cqe_bcnt = be32_to_cpu(cqe->byte_cnt);
 
-	skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt);
+	skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe,
+			      mlx5e_skb_from_cqe_linear,
+			      mlx5e_skb_from_cqe_nonlinear,
+			      rq, cqe, wi, cqe_bcnt);
 	if (!skb) {
 		/* probably for XDP */
 		if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) {
@@ -1279,8 +1283,10 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 
 	cqe_bcnt = mpwrq_get_cqe_byte_cnt(cqe);
 
-	skb = rq->mpwqe.skb_from_cqe_mpwrq(rq, wi, cqe_bcnt, head_offset,
-					   page_idx);
+	skb = INDIRECT_CALL_2(rq->mpwqe.skb_from_cqe_mpwrq,
+			      mlx5e_skb_from_cqe_mpwrq_linear,
+			      mlx5e_skb_from_cqe_mpwrq_nonlinear,
+			      rq, wi, cqe_bcnt, head_offset, page_idx);
 	if (!skb)
 		goto mpwrq_cqe_out;
 
@@ -1437,7 +1443,10 @@ void mlx5i_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 	wi       = get_frag(rq, ci);
 	cqe_bcnt = be32_to_cpu(cqe->byte_cnt);
 
-	skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt);
+	skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe,
+			      mlx5e_skb_from_cqe_linear,
+			      mlx5e_skb_from_cqe_nonlinear,
+			      rq, cqe, wi, cqe_bcnt);
 	if (!skb)
 		goto wq_free_wqe;
 
@@ -1469,7 +1478,10 @@ void mlx5e_ipsec_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
 	wi       = get_frag(rq, ci);
 	cqe_bcnt = be32_to_cpu(cqe->byte_cnt);
 
-	skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt);
+	skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe,
+			      mlx5e_skb_from_cqe_linear,
+			      mlx5e_skb_from_cqe_nonlinear,
+			      rq, cqe, wi, cqe_bcnt);
 	if (unlikely(!skb)) {
 		/* a DROP, save the page-reuse checks */
 		mlx5e_free_rx_wqe(rq, wi, true);
Signature · 1 line
2.20.1
author maintainer
#3 lore raw

PATCH v3 2/2 net/mlx5e: use indirect calls wrapper for the rx packet handler

In reply to Paolo Abeni

We can avoid another indirect call per packet wrapping the rx handler call with the proper helper.

To ensure that even the last listed direct call experience measurable gain, despite the additional conditionals we must traverse before reaching it, I tested reversing the order of the listed options, with performance differences below noise level.

Together with the previous indirect call patch, this gives ~6% performance improvement in raw UDP tput.

v2 -> v3:
- use only the direct calls always available regardless of the mlx5 build options
- drop the direct call list macro, to keep the code as simple as possible for future rework

v1 -> v2:
- update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansion

---
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +2 -1
index 0fe5f13d07cc..935bf62eddc1 100644
@@ -1333,7 +1333,8 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
 
 		mlx5_cqwq_pop(cqwq);
 
-		rq->handle_rx_cqe(rq, cqe);
+		INDIRECT_CALL_2(rq->handle_rx_cqe, mlx5e_handle_rx_cqe_mpwrq,
+				mlx5e_handle_rx_cqe, rq, cqe);
 	} while ((++work_done < budget) && (cqe = mlx5_cqwq_get_cqe(cqwq)));
 
 out:
Signature · 1 line
2.20.1
Acked-by on v3 2/2
#4 lore raw

Re: [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler

In reply to Paolo Abeni
Quoted 17 lines · Paolo Abeni

We can avoid another indirect call per packet wrapping the rx handler call with the proper helper.

To ensure that even the last listed direct call experience measurable gain, despite the additional conditionals we must traverse before reaching it, I tested reversing the order of the listed options, with performance differences below noise level.

Together with the previous indirect call patch, this gives ~6% performance improvement in raw UDP tput.

v2 -> v3:
- use only the direct calls always available regardless of the mlx5 build options
- drop the direct call list macro, to keep the code as simple as possible for future rework

Quoted 7 lines

v1 -> v2:
- update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansion

maintainer Applied
#5 lore raw
In reply to Paolo Abeni

From: Paolo Abeni <[email protected]>
Date: Wed, 12 Jun 2019 12:18:34 +0200

Quoted 15 lines

The mlx5_core driver uses several indirect calls in fast-path, some of them are invoked on each ingress packet, even for the XDP-only traffic.

This series leverage the indirect call wrappers infrastructure the avoid the expansive RETPOLINE overhead for 2 indirect calls in fast-path.

Each call is addressed on a different patch, plus we need to introduce a couple of additional helpers to cope with the higher number of possible direct-call alternatives.

v2 -> v3:
- do not add more INDIRECT_CALL_* macros
- use only the direct calls always available regardless of the mlx5 build options in the last patch

Quoted 6 lines

v1 -> v2:
- update the direct call list and use a macro to define it,
as per Saeed suggestion. An intermediated additional
macro is needed to allow arg list expansion
- patch 2/3 is unchanged, as the generated code looks better this way than with possible alternative (dropping BP hits)

Applied, thanks.