net/mlx5: use indirect call wrappers
Paolo Abeni
maintainerv3 cover net/mlx5: use indirect call wrappers
The mlx5_core driver uses several indirect calls in fast-path, some of them are invoked on each ingress packet, even for the XDP-only traffic.
This series leverage the indirect call wrappers infrastructure the avoid the expansive RETPOLINE overhead for 2 indirect calls in fast-path.
Each call is addressed on a different patch, plus we need to introduce a couple of additional helpers to cope with the higher number of possible direct-call alternatives.
v2 -> v3:
- do not add more INDIRECT_CALL_* macros
- use only the direct calls always available regardless of the mlx5 build options in the last patch
v1 -> v2:
- update the direct call list and use a macro to define it,
as per Saeed suggestion. An intermediated additional
macro is needed to allow arg list expansion
- patch 2/3 is unchanged, as the generated code looks better this way than with possible alternative (dropping BP hits)
Paolo Abeni (2):
net/mlx5e: use indirect calls wrapper for skb allocation net/mlx5e: use indirect calls wrapper for the rx packet handler
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 25 ++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v3 1/2 net/mlx5e: use indirect calls wrapper for skb allocation
We can avoid an indirect call per packet wrapping the skb creation with the appropriate helper.
--- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 22 ++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +17 -5
@@ -34,6 +34,7 @@#include <linux/ip.h> #include <linux/ipv6.h> #include <linux/tcp.h> +#include <linux/indirect_call_wrapper.h> #include <net/ip6_checksum.h> #include <net/page_pool.h> #include <net/inet_ecn.h>
@@ -1092,7 +1093,10 @@ void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi =get_frag (rq, ci); cqe_bcnt =be32_to_cpu (cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe (rq, cqe, wi, cqe_bcnt); + skb =INDIRECT_CALL_2 (rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt);if (!skb) {/* probably for XDP */ if (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags)) {
@@ -1279,8 +1283,10 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) cqe_bcnt =mpwrq_get_cqe_byte_cnt (cqe); - skb = rq->mpwqe.skb_from_cqe_mpwrq (rq, wi, cqe_bcnt, head_offset, - page_idx); + skb =INDIRECT_CALL_2 (rq->mpwqe.skb_from_cqe_mpwrq, + mlx5e_skb_from_cqe_mpwrq_linear, + mlx5e_skb_from_cqe_mpwrq_nonlinear, + rq, wi, cqe_bcnt, head_offset, page_idx);if (!skb) goto mpwrq_cqe_out;
@@ -1437,7 +1443,10 @@ void mlx5i_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi =get_frag (rq, ci); cqe_bcnt =be32_to_cpu (cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe (rq, cqe, wi, cqe_bcnt); + skb =INDIRECT_CALL_2 (rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt);if (!skb) goto wq_free_wqe;
@@ -1469,7 +1478,10 @@ void mlx5e_ipsec_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi =get_frag (rq, ci); cqe_bcnt =be32_to_cpu (cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe (rq, cqe, wi, cqe_bcnt); + skb =INDIRECT_CALL_2 (rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt);if (unlikely (!skb)) {/* a DROP, save the page-reuse checks */ mlx5e_free_rx_wqe (rq, wi, true);
Signature · 1 line
2.20.1
Re: [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation
Quoted 4 lines · Paolo Abeni
We can avoid an indirect call per packet wrapping the skb creation with the appropriate helper.
Signed-off-by Paolo Abeni <[email protected]>
Paolo Abeni
maintainerPATCH v3 2/2 net/mlx5e: use indirect calls wrapper for the rx packet handler
In reply to Paolo AbeniWe can avoid another indirect call per packet wrapping the rx handler call with the proper helper.
To ensure that even the last listed direct call experience measurable gain, despite the additional conditionals we must traverse before reaching it, I tested reversing the order of the listed options, with performance differences below noise level.
Together with the previous indirect call patch, this gives ~6% performance improvement in raw UDP tput.
v2 -> v3:
- use only the direct calls always available regardless of the mlx5 build options
- drop the direct call list macro, to keep the code as simple as possible for future rework
v1 -> v2:
- update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansion
--- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +2 -1
@@ -1333,7 +1333,8 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)mlx5_cqwq_pop (cqwq); - rq->handle_rx_cqe (rq, cqe); +INDIRECT_CALL_2 (rq->handle_rx_cqe, mlx5e_handle_rx_cqe_mpwrq, + mlx5e_handle_rx_cqe, rq, cqe); }while ((++work_done < budget) && (cqe =mlx5_cqwq_get_cqe (cqwq))); out:
Signature · 1 line
2.20.1
Re: [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler
Quoted 17 lines · Paolo Abeni
We can avoid another indirect call per packet wrapping the rx handler call with the proper helper.
To ensure that even the last listed direct call experience measurable gain, despite the additional conditionals we must traverse before reaching it, I tested reversing the order of the listed options, with performance differences below noise level.
Together with the previous indirect call patch, this gives ~6% performance improvement in raw UDP tput.
v2 -> v3:
- use only the direct calls always available regardless of the mlx5 build options
- drop the direct call list macro, to keep the code as simple as possible for future rework
Quoted 7 lines
v1 -> v2:
- update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansionSigned-off-by Paolo Abeni <[email protected]>
From: Paolo Abeni <[email protected]>
Date: Wed, 12 Jun 2019 12:18:34 +0200
Quoted 15 lines
The
mlx5_coredriver uses several indirect calls in fast-path, some of them are invoked on each ingress packet, even for the XDP-only traffic.This series leverage the indirect call wrappers infrastructure the avoid the expansive RETPOLINE overhead for 2 indirect calls in fast-path.
Each call is addressed on a different patch, plus we need to introduce a couple of additional helpers to cope with the higher number of possible direct-call alternatives.
v2 -> v3:
- do not add more INDIRECT_CALL_* macros
- use only the direct calls always available regardless of the mlx5 build options in the last patch
Quoted 6 lines
v1 -> v2:
- update the direct call list and use a macro to define it,
as per Saeed suggestion. An intermediated additional
macro is needed to allow arg list expansion
- patch 2/3 is unchanged, as the generated code looks better this way than with possible alternative (dropping BP hits)
Applied, thanks.