Skip to contentSkip to conversation

mlx5 kTLS RX offload support 2020-05-29

GIT PULL In review Saeed Mahameed 4 participants 29 messages May 29, 2020 – Jun 4, 2020

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

Hi Dave/Jakub

This series adds kTLS rx offloads support to mlx5
For more information please see tag log below.

Please pull and let me know if there is any problem.

Thanks,
Saeed.

---
The following changes since commit 394f9ebf92c899b42207d4e71465869656981ba1:

Merge branch 'hns3-next' (2020-05-28 16:39:04 -0700)

are available in the Git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2020-05-29

for you to fetch changes up to 9956363818e903ac8ef1e2d6168e8d0cc19f155b:

net/mlx5e: kTLS, Improve rx handler function call (2020-05-29 12:39:03 -0700)

----------------------------------------------------------------
mlx5-updates-2020-05-29

mlx5 kTLS rx offload:

1) Improve hardware layouts and structure for kTLS support

2) Generalize ICOSQ (Internal Channel Operations Send Queue) Due to the asynchronous nature of adding new kTLS flows and handling HW asynchronous kTLS resync requests, the XSK ICOSQ was extended to support generic async operations, such as kTLS add flow and resync, in addition to the existing XSK usages.

3) kTLS hardware flow steering and classification:
The driver already has the means to classify TCP ipv4/6 flows to send them to the corresponding RSS HW engine, as reflected in patches 3 through 5, the series will add a steering layer that will hook to the driver's TCP classifiers and will match on well known kTLS connection, in case of a match traffic will be redirected to the kTLS decryption engine, otherwise traffic will continue flowing normally to the TCP RSS engine.

3) kTLS add flow RX HW offload support
New offload contexts post their static/progress params WQEs (Work Queue Element) to communicate the newly added kTLS contexts over the per-channel async ICOSQ.

The Channel/RQ is selected according to the socket's rxq index.

A new TLS-RX workqueue is used to allow asynchronous addition of steering rules, out of the NAPI context.
It will be also used in a downstream patch in the resync procedure.

Feature is OFF by default. Can be turned on by:

$ ethtool -K <if> tls-hw-rx-offload on

4) Added mlx5 kTLS sw stats and new counters are documented in Documentation/networking/tls-offload.rst
rx_tls_ctx - number of TLS RX HW offload contexts added to device for decryption.

rx_tls_ooo - number of RX packets which were part of a TLS stream but did not arrive in the expected order and triggered the resync procedure.

rx_tls_del - number of TLS RX HW offload contexts deleted from device (connection has finished).

rx_tls_err - number of RX packets which were part of a TLS stream
but were not decrypted due to unexpected error in the state machine.

5) Implement the RX resync procedure.
The HW offload of TLS decryption in RX side might get out-of-sync due to out-of-order reception of packets.
This requires SW intervention to update the HW context and get it back in-sync.

Performance:

    CPU: Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz, 24 cores, HT off
    NIC: ConnectX-6 Dx 100GbE dual port

    Goodput (app-layer throughput) comparison:
    +---------------+-------+-------+---------+
    | # connections |   1   |   4   |    8    |
    +---------------+-------+-------+---------+
    | SW (Gbps)     |  7.26 | 24.70 |   50.30 |
    +---------------+-------+-------+---------+
    | HW (Gbps)     | 18.50 | 64.30 |   92.90 |
    +---------------+-------+-------+---------+
    | Speedup       | 2.55x | 2.56x | 1.85x * |
    +---------------+-------+-------+---------+

    * After linerate is reached, diff is observed in CPU util

----------------------------------------------------------------
Boris Pismenny (1):
net/mlx5e: Receive flow steering framework for accelerated TCP flows

Saeed Mahameed (2):
net/mlx5e: API to manipulate TTC rules destinations net/mlx5e: kTLS, Improve rx handler function call

Tariq Toukan (8):
net/mlx5: kTLS, Improve TLS params layout structures net/mlx5e: Turn XSK ICOSQ into a general asynchronous one net/mlx5e: Accel, Expose flow steering API for rules add/del net/mlx5e: kTLS, Improve TLS feature modularity net/mlx5e: kTLS, Use kernel API to extract private offload context net/mlx5e: kTLS, Add kTLS RX HW offload support net/mlx5e: kTLS, Add kTLS RX stats
net/mlx5e: kTLS, Add kTLS RX resync support

 Documentation/networking/tls-offload.rst           |   8 +
 drivers/net/ethernet/mellanox/mlx5/core/Makefile   |   3 +-
 .../net/ethernet/mellanox/mlx5/core/accel/tls.h    |  19 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h       |  22 +-
 drivers/net/ethernet/mellanox/mlx5/core/en/fs.h    |  26 +-
 drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h  |  14 +-
 .../net/ethernet/mellanox/mlx5/core/en/xsk/setup.c |  46 +-
 .../net/ethernet/mellanox/mlx5/core/en/xsk/tx.c    |  12 +-
 .../mellanox/mlx5/core/en_accel/en_accel.h         |  20 +
 .../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c  | 389 +++++++++++++
 .../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h  |  27 +
 .../ethernet/mellanox/mlx5/core/en_accel/ktls.c    | 123 ++--
 .../ethernet/mellanox/mlx5/core/en_accel/ktls.h    | 114 +---
 .../ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c | 640 +++++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c | 206 ++++---
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c        | 119 ++++
 .../mellanox/mlx5/core/en_accel/ktls_txrx.h        |  44 ++
 .../mellanox/mlx5/core/en_accel/ktls_utils.h       |  87 +++
 .../net/ethernet/mellanox/mlx5/core/en_accel/tls.c |  20 +-
 .../net/ethernet/mellanox/mlx5/core/en_accel/tls.h |   8 +-
 .../mellanox/mlx5/core/en_accel/tls_rxtx.c         |  34 +-
 .../mellanox/mlx5/core/en_accel/tls_rxtx.h         |  39 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c  |  34 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_fs.c    |  84 ++-
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c  |  33 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c    |  42 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_stats.c |  24 +
 drivers/net/ethernet/mellanox/mlx5/core/en_stats.h |  15 +
 drivers/net/ethernet/mellanox/mlx5/core/en_tx.c    |   1 -
 drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c  |  12 +-
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.c  |   4 +-
 drivers/net/ethernet/mellanox/mlx5/core/fw.c       |   3 +-
 include/linux/mlx5/device.h                        |   9 +
 include/linux/mlx5/mlx5_ifc.h                      |   5 +-
 include/linux/mlx5/qp.h                            |   2 +-
 35 files changed, 1867 insertions(+), 421 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
#1 lore raw

PATCH 1/11 net/mlx5: kTLS, Improve TLS params layout structures

From: Tariq Toukan <[email protected]>

Add explicit WQE segment structures for the TLS static and progress params.
According to the HW spec, TISN is not part of the progress params context, take it out of it.
Rename the control segment tisn field as it could hold either a TIS or a TIR number.

---
 drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h  |  2 +-
 .../ethernet/mellanox/mlx5/core/en_accel/ktls.h    |  2 +-
 .../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c | 14 +++++++++-----
 .../mellanox/mlx5/core/en_accel/tls_rxtx.c         |  2 +-
 include/linux/mlx5/device.h                        |  9 +++++++++
 include/linux/mlx5/mlx5_ifc.h                      |  5 +----
 include/linux/mlx5/qp.h                            |  2 +-
 7 files changed, 23 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +1 -1
index bfd3e1161bc66..31cac239563de 100644
@@ -182,7 +182,7 @@ mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
 
 static inline bool mlx5e_transport_inline_tx_wqe(struct mlx5_wqe_ctrl_seg *cseg)
 {
-	return cseg && !!cseg->tisn;
+	return cseg && !!cseg->tis_tir_num;
 }
 
 static inline u8
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h +1 -1
index c6180892cfcba..806ed185dd4cf 100644
@@ -19,7 +19,7 @@
 
 #define MLX5E_KTLS_PROGRESS_WQE_SZ \
 	(offsetof(struct mlx5e_tx_wqe, tls_progress_params_ctx) + \
-	 MLX5_ST_SZ_BYTES(tls_progress_params))
+	 sizeof(struct mlx5_wqe_tls_progress_params_seg))
 #define MLX5E_KTLS_PROGRESS_WQEBBS \
 	(DIV_ROUND_UP(MLX5E_KTLS_PROGRESS_WQE_SZ, MLX5_SEND_WQE_BB))
 
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +9 -5
index 3cd78d9503c12..ad7300f198157 100644
@@ -64,7 +64,7 @@ build_static_params(struct mlx5e_umr_wqe *wqe, u16 pc, u32 sqn,
 	cseg->qpn_ds           = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
 					     STATIC_PARAMS_DS_CNT);
 	cseg->fm_ce_se         = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
-	cseg->tisn             = cpu_to_be32(priv_tx->tisn << 8);
+	cseg->tis_tir_num      = cpu_to_be32(priv_tx->tisn << 8);
 
 	ucseg->flags = MLX5_UMR_INLINE;
 	ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
@@ -75,10 +75,14 @@ build_static_params(struct mlx5e_umr_wqe *wqe, u16 pc, u32 sqn,
 static void
 fill_progress_params_ctx(void *ctx, struct mlx5e_ktls_offload_context_tx *priv_tx)
 {
-	MLX5_SET(tls_progress_params, ctx, tisn, priv_tx->tisn);
-	MLX5_SET(tls_progress_params, ctx, record_tracker_state,
+	struct mlx5_wqe_tls_progress_params_seg *params;
+
+	params = ctx;
+
+	params->tis_tir_num = cpu_to_be32(priv_tx->tisn);
+	MLX5_SET(tls_progress_params, params->ctx, record_tracker_state,
 		 MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
-	MLX5_SET(tls_progress_params, ctx, auth_state,
+	MLX5_SET(tls_progress_params, params->ctx, auth_state,
 		 MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD);
 }
 
@@ -284,7 +288,7 @@ tx_post_resync_dump(struct mlx5e_txqsq *sq, skb_frag_t *frag, u32 tisn, bool fir
 
 	cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8)  | MLX5_OPCODE_DUMP);
 	cseg->qpn_ds           = cpu_to_be32((sq->sqn << 8) | ds_cnt);
-	cseg->tisn             = cpu_to_be32(tisn << 8);
+	cseg->tis_tir_num      = cpu_to_be32(tisn << 8);
 	cseg->fm_ce_se         = first ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
 
 	fsz = skb_frag_size(frag);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c +1 -1
index 05454a843b289..72d26fbc8d5bf 100644
@@ -305,7 +305,7 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
 void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
 			     struct mlx5e_accel_tx_tls_state *state)
 {
-	cseg->tisn = cpu_to_be32(state->tls_tisn << 8);
+	cseg->tis_tir_num = cpu_to_be32(state->tls_tisn << 8);
 }
 
 static int tls_update_resync_sn(struct net_device *netdev,
include/linux/mlx5/device.h +9 -0
index 1bc27aca648bb..57db125e58021 100644
@@ -458,6 +458,15 @@ enum {
 	MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS = 0x2,
 };
 
+struct mlx5_wqe_tls_static_params_seg {
+	u8     ctx[MLX5_ST_SZ_BYTES(tls_static_params)];
+};
+
+struct mlx5_wqe_tls_progress_params_seg {
+	__be32 tis_tir_num;
+	u8     ctx[MLX5_ST_SZ_BYTES(tls_progress_params)];
+};
+
 enum {
 	MLX5_SET_PORT_RESET_QKEY	= 0,
 	MLX5_SET_PORT_GUID0		= 16,
include/linux/mlx5/mlx5_ifc.h +1 -4
index fb243848132d5..e9c3c09f974f3 100644
@@ -10631,16 +10631,13 @@ struct mlx5_ifc_tls_static_params_bits {
 };
 
 struct mlx5_ifc_tls_progress_params_bits {
-	u8         reserved_at_0[0x8];
-	u8         tisn[0x18];
-
 	u8         next_record_tcp_sn[0x20];
 
 	u8         hw_resync_tcp_sn[0x20];
 
 	u8         record_tracker_state[0x2];
 	u8         auth_state[0x2];
-	u8         reserved_at_64[0x4];
+	u8         reserved_at_44[0x4];
 	u8         hw_offset_record_number[0x18];
 };
 
include/linux/mlx5/qp.h +1 -1
index f23eb18526fe7..a635b38e48559 100644
@@ -208,7 +208,7 @@ struct mlx5_wqe_ctrl_seg {
 		__be32		general_id;
 		__be32		imm;
 		__be32		umr_mkey;
-		__be32		tisn;
+		__be32		tis_tir_num;
 	};
 };
 
Signature · 1 line
2.26.2
#2 lore raw

PATCH 2/11 net/mlx5e: Turn XSK ICOSQ into a general asynchronous one

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

There is an upcoming demand (in downstream patches) for an ICOSQ to be populated out of the NAPI context, asynchronously.

There is already an existing one serving XSK-related use case. In this patch, promote this ICOSQ to serve as general async ICOSQ, to be used for XSK and non-XSK flows.

As part of this, the reg_umr bit of the SQ context is now set (if capable), as the general async ICOSQ should support possible posts of UMR WQEs.

---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  8 ++--
 .../mellanox/mlx5/core/en/xsk/setup.c         | 46 ++-----------------
 .../ethernet/mellanox/mlx5/core/en/xsk/tx.c   | 12 ++---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 24 +++++++++-
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 12 ++---
 5 files changed, 42 insertions(+), 60 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +5 -3
index c5f8a3925b168..6937aaf24235b 100644
@@ -651,9 +651,11 @@ struct mlx5e_channel {
 	/* AF_XDP zero-copy */
 	struct mlx5e_rq            xskrq;
 	struct mlx5e_xdpsq         xsksq;
-	struct mlx5e_icosq         xskicosq;
-	/* xskicosq can be accessed from any CPU - the spinlock protects it. */
-	spinlock_t                 xskicosq_lock;
+
+	/* Async ICOSQ */
+	struct mlx5e_icosq         async_icosq;
+	/* async_icosq can be accessed from any CPU - the spinlock protects it. */
+	spinlock_t                 async_icosq_lock;
 
 	/* data path - accessed per napi poll */
 	struct irq_desc *irq_desc;
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c +3 -43

100 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +6 -6
index 83dce9cdb8c2f..e0b3c61af93ea 100644
@@ -26,19 +26,19 @@ int mlx5e_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags)
 		return -ENXIO;
 
 	if (!napi_if_scheduled_mark_missed(&c->napi)) {
-		/* To avoid WQE overrun, don't post a NOP if XSKICOSQ is not
+		/* To avoid WQE overrun, don't post a NOP if async_icosq is not
 		 * active and not polled by NAPI. Return 0, because the upcoming
 		 * activate will trigger the IRQ for us.
 		 */
-		if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &c->xskicosq.state)))
+		if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &c->async_icosq.state)))
 			return 0;
 
-		if (test_and_set_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->xskicosq.state))
+		if (test_and_set_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state))
 			return 0;
 
-		spin_lock(&c->xskicosq_lock);
-		mlx5e_trigger_irq(&c->xskicosq);
-		spin_unlock(&c->xskicosq_lock);
+		spin_lock(&c->async_icosq_lock);
+		mlx5e_trigger_irq(&c->async_icosq);
+		spin_unlock(&c->async_icosq_lock);
 	}
 
 	return 0;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +22 -2
index 2c36a04181b84..db18f602cdb2f 100644
@@ -1817,10 +1817,14 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
 	struct dim_cq_moder icocq_moder = {0, 0};
 	int err;
 
-	err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
+	err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->async_icosq.cq);
 	if (err)
 		return err;
 
+	err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
+	if (err)
+		goto err_close_async_icosq_cq;
+
 	err = mlx5e_open_tx_cqs(c, params, cparam);
 	if (err)
 		goto err_close_icosq_cq;
@@ -1841,10 +1845,16 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
 
 	napi_enable(&c->napi);
 
-	err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->icosq);
+	spin_lock_init(&c->async_icosq_lock);
+
+	err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->async_icosq);
 	if (err)
 		goto err_disable_napi;
 
+	err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->icosq);
+	if (err)
+		goto err_close_async_icosq;
+
 	err = mlx5e_open_sqs(c, params, cparam);
 	if (err)
 		goto err_close_icosq;
@@ -1879,6 +1889,9 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
 err_close_icosq:
 	mlx5e_close_icosq(&c->icosq);
 
+err_close_async_icosq:
+	mlx5e_close_icosq(&c->async_icosq);
+
 err_disable_napi:
 	napi_disable(&c->napi);
 
@@ -1897,6 +1910,9 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
 err_close_icosq_cq:
 	mlx5e_close_cq(&c->icosq.cq);
 
+err_close_async_icosq_cq:
+	mlx5e_close_cq(&c->async_icosq.cq);
+
 	return err;
 }
 
@@ -1908,6 +1924,7 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
 		mlx5e_close_xdpsq(&c->rq_xdpsq);
 	mlx5e_close_sqs(c);
 	mlx5e_close_icosq(&c->icosq);
+	mlx5e_close_icosq(&c->async_icosq);
 	napi_disable(&c->napi);
 	if (c->xdp)
 		mlx5e_close_cq(&c->rq_xdpsq.cq);
@@ -1915,6 +1932,7 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
 	mlx5e_close_cq(&c->xdpsq.cq);
 	mlx5e_close_tx_cqs(c);
 	mlx5e_close_cq(&c->icosq.cq);
+	mlx5e_close_cq(&c->async_icosq.cq);
 }
 
 static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)
@@ -1995,6 +2013,7 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
 	for (tc = 0; tc < c->num_tc; tc++)
 		mlx5e_activate_txqsq(&c->sq[tc]);
 	mlx5e_activate_icosq(&c->icosq);
+	mlx5e_activate_icosq(&c->async_icosq);
 	mlx5e_activate_rq(&c->rq);
 
 	if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
@@ -2009,6 +2028,7 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
 		mlx5e_deactivate_xsk(c);
 
 	mlx5e_deactivate_rq(&c->rq);
+	mlx5e_deactivate_icosq(&c->async_icosq);
 	mlx5e_deactivate_icosq(&c->icosq);
 	for (tc = 0; tc < c->num_tc; tc++)
 		mlx5e_deactivate_txqsq(&c->sq[tc]);
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +6 -6
index 8480278f2ee20..ad99f270efb5e 100644
@@ -149,17 +149,17 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	}
 
 	mlx5e_poll_ico_cq(&c->icosq.cq);
+	if (mlx5e_poll_ico_cq(&c->async_icosq.cq))
+		/* Don't clear the flag if nothing was polled to prevent
+		 * queueing more WQEs and overflowing XSKICOSQ.
+		 */
+		clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state);
 
 	busy |= INDIRECT_CALL_2(rq->post_wqes,
 				mlx5e_post_rx_mpwqes,
 				mlx5e_post_rx_wqes,
 				rq);
 	if (xsk_open) {
-		if (mlx5e_poll_ico_cq(&c->xskicosq.cq))
-			/* Don't clear the flag if nothing was polled to prevent
-			 * queueing more WQEs and overflowing XSKICOSQ.
-			 */
-			clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->xskicosq.state);
 		busy |= mlx5e_poll_xdpsq_cq(&xsksq->cq);
 		busy_xsk |= mlx5e_napi_xsk_post(xsksq, xskrq);
 	}
@@ -189,11 +189,11 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 
 	mlx5e_cq_arm(&rq->cq);
 	mlx5e_cq_arm(&c->icosq.cq);
+	mlx5e_cq_arm(&c->async_icosq.cq);
 	mlx5e_cq_arm(&c->xdpsq.cq);
 
 	if (xsk_open) {
 		mlx5e_handle_rx_dim(xskrq);
-		mlx5e_cq_arm(&c->xskicosq.cq);
 		mlx5e_cq_arm(&xsksq->cq);
 		mlx5e_cq_arm(&xskrq->cq);
 	}
Signature · 1 line
2.26.2
#3 lore raw

PATCH 3/11 net/mlx5e: API to manipulate TTC rules destinations

In reply to Saeed Mahameed

Store the default destinations of the on-load generated TTC (Traffic Type Classifier) rules in the ttc rules table.

Introduce TTC API functions to manipulate/restore and get the TTC rule destination and use these API functions in arfs implementation.

This will allow a better decoupling between TTC implementation and its users.

---
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   | 16 +++-
 .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 34 +++-----
 .../net/ethernet/mellanox/mlx5/core/en_fs.c   | 84 ++++++++++++++-----
 3 files changed, 86 insertions(+), 48 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +13 -3
index 0416f77121096..c633579474c34 100644
@@ -105,11 +105,16 @@ enum mlx5e_tunnel_types {
 
 bool mlx5e_tunnel_inner_ft_supported(struct mlx5_core_dev *mdev);
 
+struct mlx5e_ttc_rule {
+	struct mlx5_flow_handle *rule;
+	struct mlx5_flow_destination default_dest;
+};
+
 /* L3/L4 traffic type classifier */
 struct mlx5e_ttc_table {
-	struct mlx5e_flow_table  ft;
-	struct mlx5_flow_handle	 *rules[MLX5E_NUM_TT];
-	struct mlx5_flow_handle  *tunnel_rules[MLX5E_NUM_TUNNEL_TT];
+	struct mlx5e_flow_table ft;
+	struct mlx5e_ttc_rule rules[MLX5E_NUM_TT];
+	struct mlx5_flow_handle *tunnel_rules[MLX5E_NUM_TUNNEL_TT];
 };
 
 /* NIC prio FTS */
@@ -248,6 +253,11 @@ void mlx5e_destroy_inner_ttc_table(struct mlx5e_priv *priv,
 				   struct mlx5e_ttc_table *ttc);
 
 void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft);
+int mlx5e_ttc_fwd_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type,
+		       struct mlx5_flow_destination *new_dest);
+struct mlx5_flow_destination
+mlx5e_ttc_get_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type);
+int mlx5e_ttc_fwd_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type);
 
 void mlx5e_enable_cvlan_filter(struct mlx5e_priv *priv);
 void mlx5e_disable_cvlan_filter(struct mlx5e_priv *priv);
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c +12 -22
index 014639ea06e34..8f1564159ea4d 100644
@@ -90,23 +90,15 @@ static enum mlx5e_traffic_types arfs_get_tt(enum arfs_type type)
 
 static int arfs_disable(struct mlx5e_priv *priv)
 {
-	struct mlx5_flow_destination dest = {};
-	struct mlx5e_tir *tir = priv->indir_tir;
-	int err = 0;
-	int tt;
-	int i;
+	int err, i;
 
-	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
 	for (i = 0; i < ARFS_NUM_TYPES; i++) {
-		dest.tir_num = tir[i].tirn;
-		tt = arfs_get_tt(i);
-		/* Modify ttc rules destination to bypass the aRFS tables*/
-		err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
-						   &dest, NULL);
+		/* Modify ttc rules destination back to their default */
+		err = mlx5e_ttc_fwd_default_dest(priv, arfs_get_tt(i));
 		if (err) {
 			netdev_err(priv->netdev,
-				   "%s: modify ttc destination failed\n",
-				   __func__);
+				   "%s: modify ttc[%d] default destination failed, err(%d)\n",
+				   __func__, arfs_get_tt(i), err);
 			return err;
 		}
 	}
@@ -125,21 +117,17 @@ int mlx5e_arfs_disable(struct mlx5e_priv *priv)
 int mlx5e_arfs_enable(struct mlx5e_priv *priv)
 {
 	struct mlx5_flow_destination dest = {};
-	int err = 0;
-	int tt;
-	int i;
+	int err, i;
 
 	dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
 	for (i = 0; i < ARFS_NUM_TYPES; i++) {
 		dest.ft = priv->fs.arfs.arfs_tables[i].ft.t;
-		tt = arfs_get_tt(i);
 		/* Modify ttc rules destination to point on the aRFS FTs */
-		err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
-						   &dest, NULL);
+		err = mlx5e_ttc_fwd_dest(priv, arfs_get_tt(i), &dest);
 		if (err) {
 			netdev_err(priv->netdev,
-				   "%s: modify ttc destination failed err=%d\n",
-				   __func__, err);
+				   "%s: modify ttc[%d] dest to arfs, failed err(%d)\n",
+				   __func__, arfs_get_tt(i), err);
 			arfs_disable(priv);
 			return err;
 		}
@@ -186,8 +174,10 @@ static int arfs_add_default_rule(struct mlx5e_priv *priv,
 		return -EINVAL;
 	}
 
+	/* FIXME: Must use mlx5e_ttc_get_default_dest(),
+	 * but can't since TTC default is not setup yet !
+	 */
 	dest.tir_num = tir[tt].tirn;
-
 	arfs_t->default_rule = mlx5_add_flow_rules(arfs_t->ft.t, NULL,
 						   &flow_act,
 						   &dest, 1);
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c +61 -23

155 lines not on this page · raw message

Signature · 1 line
2.26.2
#4 lore raw

PATCH 4/11 net/mlx5e: Receive flow steering framework for accelerated TCP flows

In reply to Saeed Mahameed

From: Boris Pismenny <[email protected]>

The framework allows creating flow tables to steer incoming traffic of TCP sockets to the acceleration TIRs.
This is used in downstream patches for TLS, and will be used in the future for other offloads.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |  10 +
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      | 275 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/fs_tcp.h      |  18 ++
 .../net/ethernet/mellanox/mlx5/core/fs_core.c |   4 +-
 5 files changed, 306 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
index b61e47bc16e84..8ffa1325a18f9 100644
@@ -74,7 +74,7 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
 				     en_accel/ipsec_stats.o
 
 mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
-				   en_accel/ktls.o en_accel/ktls_tx.o
+				   en_accel/ktls.o en_accel/ktls_tx.o en_accel/fs_tcp.o
 
 mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
 					steering/dr_matcher.o steering/dr_rule.o \
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +10 -0
index c633579474c34..385cbff1caf1d 100644
@@ -123,6 +123,9 @@ enum {
 	MLX5E_L2_FT_LEVEL,
 	MLX5E_TTC_FT_LEVEL,
 	MLX5E_INNER_TTC_FT_LEVEL,
+#ifdef CONFIG_MLX5_EN_TLS
+	MLX5E_ACCEL_FS_TCP_FT_LEVEL,
+#endif
 #ifdef CONFIG_MLX5_EN_ARFS
 	MLX5E_ARFS_FT_LEVEL
 #endif
@@ -216,6 +219,10 @@ static inline int mlx5e_arfs_enable(struct mlx5e_priv *priv) { return -EOPNOTSUP
 static inline int mlx5e_arfs_disable(struct mlx5e_priv *priv) {	return -EOPNOTSUPP; }
 #endif
 
+#ifdef CONFIG_MLX5_EN_TLS
+struct mlx5e_accel_fs_tcp;
+#endif
+
 struct mlx5e_flow_steering {
 	struct mlx5_flow_namespace      *ns;
 #ifdef CONFIG_MLX5_EN_RXNFC
@@ -229,6 +236,9 @@ struct mlx5e_flow_steering {
 #ifdef CONFIG_MLX5_EN_ARFS
 	struct mlx5e_arfs_tables        arfs;
 #endif
+#ifdef CONFIG_MLX5_EN_TLS
+	struct mlx5e_accel_fs_tcp      *accel_tcp;
+#endif
 };
 
 struct ttc_params {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c new file +275 -0

275 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h new file +18 -0
new file mode 100644
index 0000000000000..0df53473550af
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#ifndef __MLX5E_ACCEL_FS_TCP_H__
+#define __MLX5E_ACCEL_FS_TCP_H__
+
+#include "en.h"
+
+#ifdef CONFIG_MLX5_EN_TLS
+int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv);
+void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv);
+#else
+static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
+static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
+#endif
+
+#endif /* __MLX5E_ACCEL_FS_TCP_H__ */
+
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +2 -2
index 4bebbecb078d2..c3c111fb1c817 100644
@@ -106,8 +106,8 @@
 #define ETHTOOL_PRIO_NUM_LEVELS 1
 #define ETHTOOL_NUM_PRIOS 11
 #define ETHTOOL_MIN_LEVEL (KERNEL_MIN_LEVEL + ETHTOOL_NUM_PRIOS)
-/* Vlan, mac, ttc, inner ttc, aRFS */
-#define KERNEL_NIC_PRIO_NUM_LEVELS 5
+/* Vlan, mac, ttc, inner ttc, {aRFS/accel} */
+#define KERNEL_NIC_PRIO_NUM_LEVELS 6
 #define KERNEL_NIC_NUM_PRIOS 1
 /* One more level for tc */
 #define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
Signature · 1 line
2.26.2
#5 lore raw

PATCH 5/11 net/mlx5e: Accel, Expose flow steering API for rules add/del

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Given a socket, the function extracts the TCP/IP{4,6} ntuple and adds rule to steering.
Another function gets the rule and deletes it.

---
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      | 114 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/fs_tcp.h      |   9 ++
 2 files changed, 123 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +114 -0
index 882ea296e7eec..c5cd96ef8a7e3 100644
@@ -26,6 +26,120 @@ static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
 	}
 }
 
+static void accel_fs_tcp_set_ipv4_flow(struct mlx5_flow_spec *spec, struct sock *sk)
+{
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ethertype);
+	MLX5_SET(fte_match_param, spec->match_value, outer_headers.ethertype, ETH_P_IP);
+	memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+			    outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4),
+	       &inet_sk(sk)->inet_daddr, 4);
+	memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+			    outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4),
+	       &inet_sk(sk)->inet_rcv_saddr, 4);
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+			 outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4);
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+			 outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4);
+}
+
+static void accel_fs_tcp_set_ipv6_flow(struct mlx5_flow_spec *spec, struct sock *sk)
+{
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ethertype);
+	MLX5_SET(fte_match_param, spec->match_value, outer_headers.ethertype, ETH_P_IPV6);
+	memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+			    outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+	       &sk->sk_v6_daddr, 16);
+	memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+			    outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+	       &inet6_sk(sk)->saddr, 16);
+	memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
+			    outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+	       0xff, 16);
+	memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
+			    outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+	       0xff, 16);
+}
+
+void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
+{
+	mlx5_del_flow_rules(rule);
+}
+
+struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+					       struct sock *sk, u32 tirn,
+					       uint32_t flow_tag)
+{
+	struct mlx5_flow_destination dest = {};
+	struct mlx5e_flow_table *ft = NULL;
+	struct mlx5e_accel_fs_tcp *fs_tcp;
+	MLX5_DECLARE_FLOW_ACT(flow_act);
+	struct mlx5_flow_handle *flow;
+	struct mlx5_flow_spec *spec;
+
+	spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
+	if (!spec)
+		return ERR_PTR(-ENOMEM);
+
+	fs_tcp = priv->fs.accel_tcp;
+
+	spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+	switch (sk->sk_family) {
+	case AF_INET:
+		accel_fs_tcp_set_ipv4_flow(spec, sk);
+		ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
+		mlx5e_dbg(HW, priv, "%s flow is %pI4:%d -> %pI4:%d\n", __func__,
+			  &inet_sk(sk)->inet_rcv_saddr,
+			  inet_sk(sk)->inet_sport,
+			  &inet_sk(sk)->inet_daddr,
+			  inet_sk(sk)->inet_dport);
+		break;
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+		if (!sk->sk_ipv6only &&
+		    ipv6_addr_type(&sk->sk_v6_daddr) == IPV6_ADDR_MAPPED) {
+			accel_fs_tcp_set_ipv4_flow(spec, sk);
+			ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
+		} else {
+			accel_fs_tcp_set_ipv6_flow(spec, sk);
+			ft = &fs_tcp->tables[ACCEL_FS_IPV6_TCP];
+		}
+		break;
+#endif
+	default:
+		break;
+	}
+
+	if (!ft) {
+		flow = ERR_PTR(-EINVAL);
+		goto out;
+	}
+
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+			 outer_headers.tcp_dport);
+	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+			 outer_headers.tcp_sport);
+	MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_dport,
+		 ntohs(inet_sk(sk)->inet_sport));
+	MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_sport,
+		 ntohs(inet_sk(sk)->inet_dport));
+
+	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+	dest.tir_num = tirn;
+	if (flow_tag != MLX5_FS_DEFAULT_FLOW_TAG) {
+		spec->flow_context.flow_tag = flow_tag;
+		spec->flow_context.flags = FLOW_CONTEXT_HAS_TAG;
+	}
+
+	flow = mlx5_add_flow_rules(ft->t, spec, &flow_act, &dest, 1);
+	if (IS_ERR(flow))
+		netdev_err(priv->netdev, "mlx5_add_flow_rules() failed, flow is %ld\n",
+			   PTR_ERR(flow));
+
+out:
+	kvfree(spec);
+	return flow;
+}
+
 static int accel_fs_tcp_add_default_rule(struct mlx5e_priv *priv,
 					 enum accel_fs_tcp_type type)
 {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h +9 -0
index 0df53473550af..4341806a7b689 100644
@@ -9,9 +9,18 @@
 #ifdef CONFIG_MLX5_EN_TLS
 int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv);
 void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv);
+struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+					       struct sock *sk, u32 tirn,
+					       uint32_t flow_tag);
+void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule);
 #else
 static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
 static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
+static inline struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+							     struct sock *sk, u32 tirn,
+							     uint32_t flow_tag)
+{ return NULL; }
+static inline void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule) {}
 #endif
 
 #endif /* __MLX5E_ACCEL_FS_TCP_H__ */
Signature · 1 line
2.26.2
#6 lore raw

PATCH 6/11 net/mlx5e: kTLS, Improve TLS feature modularity

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Better separate the code into c/h files, so that kTLS internals are exposed to the corresponding non-accel flow as follows:
- Necessary datapath functions are exposed via ktls_txrx.h.
- Necessary caps and configuration functions are exposed via ktls.h, which became very small.

In addition, kTLS internal code sharing is done via ktls_utils.h, which is not exposed to any non-accel file.

Add explicit WQE structures for the TLS static and progress params, breaking the union of the static with UMR, and the progress with PSV.

Generalize the API as a preparation for TLS RX offload support.

Move kTLS TX-specific code to the proper file.
Remove the inline tag for function in C files, let the compiler decide. Use kzalloc/kfree for the priv_tx context.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   3 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  14 +-
 .../mellanox/mlx5/core/en_accel/ktls.c        |  62 +----
 .../mellanox/mlx5/core/en_accel/ktls.h        | 107 ---------
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     | 214 +++++++++++-------
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c   | 110 +++++++++
 .../mellanox/mlx5/core/en_accel/ktls_txrx.h   |  38 ++++
 .../mellanox/mlx5/core/en_accel/ktls_utils.h  |  69 ++++++
 .../mellanox/mlx5/core/en_accel/tls.c         |  14 --
 .../mellanox/mlx5/core/en_accel/tls.h         |   7 -
 .../mellanox/mlx5/core/en_accel/tls_rxtx.c    |  14 ++
 .../mellanox/mlx5/core/en_accel/tls_rxtx.h    |  13 +-
 .../net/ethernet/mellanox/mlx5/core/en_tx.c   |   1 -
 13 files changed, 378 insertions(+), 288 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -1
index 8ffa1325a18f9..70ad24fff2e23 100644
@@ -74,7 +74,8 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
 				     en_accel/ipsec_stats.o
 
 mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
-				   en_accel/ktls.o en_accel/ktls_tx.o en_accel/fs_tcp.o
+				   en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
+				   en_accel/ktls_tx.o
 
 mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
 					steering/dr_matcher.o steering/dr_rule.o \
drivers/net/ethernet/mellanox/mlx5/core/en.h +3 -11
index 6937aaf24235b..55b89d3900f21 100644
@@ -191,13 +191,8 @@ static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
 
 struct mlx5e_tx_wqe {
 	struct mlx5_wqe_ctrl_seg ctrl;
-	union {
-		struct {
-			struct mlx5_wqe_eth_seg  eth;
-			struct mlx5_wqe_data_seg data[0];
-		};
-		u8 tls_progress_params_ctx[0];
-	};
+	struct mlx5_wqe_eth_seg  eth;
+	struct mlx5_wqe_data_seg data[0];
 };
 
 struct mlx5e_rx_wqe_ll {
@@ -213,10 +208,7 @@ struct mlx5e_umr_wqe {
 	struct mlx5_wqe_ctrl_seg       ctrl;
 	struct mlx5_wqe_umr_ctrl_seg   uctrl;
 	struct mlx5_mkey_seg           mkc;
-	union {
-		struct mlx5_mtt        inline_mtts[0];
-		u8                     tls_static_params_ctx[0];
-	};
+	struct mlx5_mtt                inline_mtts[0];
 };
 
 extern const char mlx5e_self_tests[][ETH_GSTRING_LEN];
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c +5 -57

88 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h +0 -107

122 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +127 -87

271 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c new file +110 -0

110 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h new file +38 -0
new file mode 100644
index 0000000000000..a325f34831176
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#ifndef __MLX5E_KTLS_TXRX_H__
+#define __MLX5E_KTLS_TXRX_H__
+
+#ifdef CONFIG_MLX5_EN_TLS
+
+#include <net/tls.h>
+#include "en.h"
+#include "en/txrx.h"
+
+struct mlx5e_accel_tx_tls_state {
+	u32 tls_tisn;
+};
+
+u16 mlx5e_ktls_get_stop_room(struct mlx5e_txqsq *sq);
+u8 mlx5e_ktls_dumps_num_wqebbs(struct mlx5e_txqsq *sq, unsigned int nfrags,
+			       unsigned int sync_len);
+
+bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
+			      struct sk_buff *skb, int datalen,
+			      struct mlx5e_accel_tx_tls_state *state);
+
+void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
+					   struct mlx5e_tx_wqe_info *wi,
+					   u32 *dma_fifo_cc);
+#else
+static inline void
+mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
+				      struct mlx5e_tx_wqe_info *wi,
+				      u32 *dma_fifo_cc)
+{
+}
+
+#endif /* CONFIG_MLX5_EN_TLS */
+
+#endif /* __MLX5E_TLS_TXRX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h new file +69 -0

69 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c +0 -14
index c27e9a609d519..fba561ffe1d4e 100644
@@ -240,17 +240,3 @@ void mlx5e_tls_cleanup(struct mlx5e_priv *priv)
 	kfree(tls);
 	priv->tls = NULL;
 }
-
-u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
-{
-	struct mlx5_core_dev *mdev = sq->channel->mdev;
-
-	if (!mlx5_accel_is_tls_device(mdev))
-		return 0;
-
-	if (MLX5_CAP_GEN(mdev, tls_tx))
-		return mlx5e_ktls_get_stop_room(sq);
-
-	/* Resync SKB. */
-	return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
-}
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h +0 -7
index 9219bdb2786e2..9015f3f7792d2 100644
@@ -94,8 +94,6 @@ int mlx5e_tls_get_count(struct mlx5e_priv *priv);
 int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data);
 int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data);
 
-u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq);
-
 #else
 
 static inline void mlx5e_tls_build_netdev(struct mlx5e_priv *priv)
@@ -110,11 +108,6 @@ static inline int mlx5e_tls_get_count(struct mlx5e_priv *priv) { return 0; }
 static inline int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data) { return 0; }
 static inline int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data) { return 0; }
 
-static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
-{
-	return 0;
-}
-
 #endif
 
 #endif /* __MLX5E_TLS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c +14 -0
index 72d26fbc8d5bf..4d796fea906d1 100644
@@ -387,3 +387,17 @@ void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 	remove_metadata_hdr(skb);
 	*cqe_bcnt -= MLX5E_METADATA_ETHER_LEN;
 }
+
+u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
+{
+	struct mlx5_core_dev *mdev = sq->channel->mdev;
+
+	if (!mlx5_accel_is_tls_device(mdev))
+		return 0;
+
+	if (MLX5_CAP_GEN(mdev, tls_tx))
+		return mlx5e_ktls_get_stop_room(sq);
+
+	/* Resync SKB. */
+	return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
+}
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h +10 -3
index a50d0394df0ae..2a7b98531539f 100644
@@ -34,15 +34,15 @@
 #ifndef __MLX5E_TLS_RXTX_H__
 #define __MLX5E_TLS_RXTX_H__
 
+#include "en_accel/ktls_txrx.h"
+
 #ifdef CONFIG_MLX5_EN_TLS
 
 #include <linux/skbuff.h>
 #include "en.h"
 #include "en/txrx.h"
 
-struct mlx5e_accel_tx_tls_state {
-	u32 tls_tisn;
-};
+u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq);
 
 bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
 			     struct sk_buff *skb, struct mlx5e_accel_tx_tls_state *state);
@@ -52,6 +52,13 @@ void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *c
 void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 			     u32 *cqe_bcnt);
 
+#else
+
+static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
+{
+	return 0;
+}
+
 #endif /* CONFIG_MLX5_EN_TLS */
 
 #endif /* __MLX5E_TLS_RXTX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +0 -1
index 6d406063aca42..da596de3abba2 100644
@@ -38,7 +38,6 @@
 #include "en/txrx.h"
 #include "ipoib/ipoib.h"
 #include "en_accel/en_accel.h"
-#include "en_accel/ktls.h"
 #include "lib/clock.h"
 
 static void mlx5e_dma_unmap_wqe_err(struct mlx5e_txqsq *sq, u8 num_dma)
Signature · 1 line
2.26.2
#7 lore raw

PATCH 7/11 net/mlx5e: kTLS, Use kernel API to extract private offload context

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Modify the implementation of the private kTLS TX HW offload context getter and setter, so it uses the kernel API functions, instead of a local shadow structure.
A single BUILD_BUG_ON check is sufficient, remove the duplicate.

---
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     | 28 ++++++-------------
 1 file changed, 9 insertions(+), 19 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +9 -19
index 349e29214b928..5a980f93c3263 100644
@@ -56,37 +56,26 @@ struct mlx5e_ktls_offload_context_tx {
 	bool ctx_post_pending;
 };
 
-struct mlx5e_ktls_offload_context_tx_shadow {
-	struct tls_offload_context_tx         tx_ctx;
-	struct mlx5e_ktls_offload_context_tx *priv_tx;
-};
-
 static void
 mlx5e_set_ktls_tx_priv_ctx(struct tls_context *tls_ctx,
 			   struct mlx5e_ktls_offload_context_tx *priv_tx)
 {
-	struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
-	struct mlx5e_ktls_offload_context_tx_shadow *shadow;
+	struct mlx5e_ktls_offload_context_tx **ctx =
+		__tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_TX);
 
-	BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
+	BUILD_BUG_ON(sizeof(struct mlx5e_ktls_offload_context_tx *) >
+		     TLS_OFFLOAD_CONTEXT_SIZE_TX);
 
-	shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
-
-	shadow->priv_tx = priv_tx;
-	priv_tx->tx_ctx = tx_ctx;
+	*ctx = priv_tx;
 }
 
 static struct mlx5e_ktls_offload_context_tx *
 mlx5e_get_ktls_tx_priv_ctx(struct tls_context *tls_ctx)
 {
-	struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
-	struct mlx5e_ktls_offload_context_tx_shadow *shadow;
-
-	BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
-
-	shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
+	struct mlx5e_ktls_offload_context_tx **ctx =
+		__tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_TX);
 
-	return shadow->priv_tx;
+	return *ctx;
 }
 
 int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
@@ -113,6 +102,7 @@ int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
 	priv_tx->expected_seq = start_offload_tcp_sn;
 	priv_tx->crypto_info  =
 		*(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
+	priv_tx->tx_ctx = tls_offload_ctx_tx(tls_ctx);
 
 	mlx5e_set_ktls_tx_priv_ctx(tls_ctx, priv_tx);
 
Signature · 1 line
2.26.2
#8 lore raw

PATCH 8/11 net/mlx5e: kTLS, Add kTLS RX HW offload support

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Implement driver support for the kTLS RX HW offload feature. Resync support is added in a downstream patch.

New offload contexts post their static/progress params WQEs over the per-channel async ICOSQ, protected under a spin-lock. The Channel/RQ is selected according to the socket's rxq index.

Feature is OFF by default. Can be turned on by:

$ ethtool -K <if> tls-hw-rx-offload on

A new TLS-RX workqueue is used to allow asynchronous addition of steering rules, out of the NAPI context.
It will be also used in a downstream patch in the resync procedure.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../ethernet/mellanox/mlx5/core/accel/tls.h   |  19 +-
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |  11 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |  20 ++
 .../mellanox/mlx5/core/en_accel/ktls.c        |  69 +++-
 .../mellanox/mlx5/core/en_accel/ktls.h        |  19 +-
 .../mellanox/mlx5/core/en_accel/ktls_rx.c     | 317 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     |   2 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c   |  20 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.h   |   4 +
 .../mellanox/mlx5/core/en_accel/ktls_utils.h  |   6 +
 .../mellanox/mlx5/core/en_accel/tls.c         |   6 +
 .../mellanox/mlx5/core/en_accel/tls.h         |   1 +
 .../mellanox/mlx5/core/en_accel/tls_rxtx.c    |  20 +-
 .../mellanox/mlx5/core/en_accel/tls_rxtx.h    |   4 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   9 +
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  33 +-
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   3 +-
 18 files changed, 534 insertions(+), 31 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
index 70ad24fff2e23..1e7c7f10db6e2 100644
@@ -75,7 +75,7 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
 
 mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
 				   en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
-				   en_accel/ktls_tx.o
+				   en_accel/ktls_tx.o en_accel/ktls_rx.o
 
 mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
 					steering/dr_matcher.o steering/dr_rule.o \
drivers/net/ethernet/mellanox/mlx5/core/accel/tls.h +18 -1
index e09bc3858d574..9f9247ddcb88d 100644
@@ -43,9 +43,20 @@ int mlx5_ktls_create_key(struct mlx5_core_dev *mdev,
 			 u32 *p_key_id);
 void mlx5_ktls_destroy_key(struct mlx5_core_dev *mdev, u32 key_id);
 
+static inline bool mlx5_accel_is_ktls_tx(struct mlx5_core_dev *mdev)
+{
+	return MLX5_CAP_GEN(mdev, tls_tx);
+}
+
+static inline bool mlx5_accel_is_ktls_rx(struct mlx5_core_dev *mdev)
+{
+	return MLX5_CAP_GEN(mdev, tls_rx);
+}
+
 static inline bool mlx5_accel_is_ktls_device(struct mlx5_core_dev *mdev)
 {
-	if (!MLX5_CAP_GEN(mdev, tls_tx))
+	if (!mlx5_accel_is_ktls_tx(mdev) &&
+	    !mlx5_accel_is_ktls_rx(mdev))
 		return false;
 
 	if (!MLX5_CAP_GEN(mdev, log_max_dek))
@@ -67,6 +78,12 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
 	return false;
 }
 #else
+static inline bool mlx5_accel_is_ktls_tx(struct mlx5_core_dev *mdev)
+{ return false; }
+
+static inline bool mlx5_accel_is_ktls_rx(struct mlx5_core_dev *mdev)
+{ return false; }
+
 static inline int
 mlx5_ktls_create_key(struct mlx5_core_dev *mdev,
 		     struct tls_crypto_info *crypto_info,
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +11 -0
index 31cac239563de..ad1525aa5670b 100644
@@ -11,6 +11,10 @@
 enum mlx5e_icosq_wqe_type {
 	MLX5E_ICOSQ_WQE_NOP,
 	MLX5E_ICOSQ_WQE_UMR_RX,
+#ifdef CONFIG_MLX5_EN_TLS
+	MLX5E_ICOSQ_WQE_UMR_TLS,
+	MLX5E_ICOSQ_WQE_SET_PSV_TLS,
+#endif
 };
 
 static inline bool
@@ -114,9 +118,16 @@ struct mlx5e_icosq_wqe_info {
 		struct {
 			struct mlx5e_rq *rq;
 		} umr;
+		struct {
+#ifdef CONFIG_MLX5_EN_TLS
+			struct mlx5e_ktls_offload_context_rx *priv_rx;
+#endif
+		} accel;
 	};
 };
 
+void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq);
+
 static inline u16 mlx5e_icosq_get_next_pi(struct mlx5e_icosq *sq, u16 size)
 {
 	struct mlx5_wq_cyc *wq = &sq->wq;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +20 -0
index fac145dcf2ceb..7b6abea850d44 100644
@@ -37,6 +37,7 @@
 #include <linux/skbuff.h>
 #include <linux/netdevice.h>
 #include "en_accel/ipsec_rxtx.h"
+#include "en_accel/tls.h"
 #include "en_accel/tls_rxtx.h"
 #include "en.h"
 #include "en/txrx.h"
@@ -147,4 +148,23 @@ static inline bool mlx5e_accel_tx_finish(struct mlx5e_priv *priv,
 	return true;
 }
 
+static inline int mlx5e_accel_sk_get_rxq(struct sock *sk)
+{
+	int rxq = sk_rx_queue_get(sk);
+
+	if (unlikely(rxq == -1))
+		rxq = 0;
+
+	return rxq;
+}
+
+static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv)
+{
+	return mlx5e_ktls_init_rx(priv);
+}
+
+static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv)
+{
+	mlx5e_ktls_cleanup_rx(priv);
+}
 #endif /* __MLX5E_EN_ACCEL_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c +59 -10

104 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h +18 -1
index 69d736954977c..baa58b62e8df0 100644
@@ -9,13 +9,30 @@
 #ifdef CONFIG_MLX5_EN_TLS
 
 void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv);
-
+int mlx5e_ktls_init_rx(struct mlx5e_priv *priv);
+void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv);
+int mlx5e_ktls_set_feature_rx(struct net_device *netdev, bool enable);
 #else
 
 static inline void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)
 {
 }
 
+static inline int mlx5e_ktls_init_rx(struct mlx5e_priv *priv)
+{
+	return 0;
+}
+
+static inline void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv)
+{
+}
+
+static inline int mlx5e_ktls_set_feature_rx(struct net_device *netdev, bool enable)
+{
+	netdev_warn(netdev, "kTLS is not supported\n");
+	return -EOPNOTSUPP;
+}
+
 #endif
 
 #endif /* __MLX5E_TLS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c new file +317 -0

317 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +1 -1
index 5a980f93c3263..9c34ffa55b328 100644
@@ -188,7 +188,7 @@ post_progress_params(struct mlx5e_txqsq *sq,
 	num_wqebbs = MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS;
 	pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
 	wqe = MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi);
-	mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_tx->tisn, fence,
+	mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_tx->tisn, fence, 0,
 					 TLS_OFFLOAD_CTX_DIR_TX);
 	tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
 	sq->pc += num_wqebbs;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c +14 -6
index edf404eaa2752..c1f1ad32ca4c2 100644
@@ -59,11 +59,13 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
 {
 	struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
 	struct mlx5_wqe_ctrl_seg     *cseg  = &wqe->ctrl;
+	u8 opmod = direction == TLS_OFFLOAD_CTX_DIR_TX ?
+		MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS :
+		MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS;
 
 #define STATIC_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
 
-	cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR |
-					     (MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS << 24));
+	cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR | (opmod << 24));
 	cseg->qpn_ds           = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
 					     STATIC_PARAMS_DS_CNT);
 	cseg->fm_ce_se         = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
@@ -76,12 +78,15 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
 }
 
 static void
-fill_progress_params(struct mlx5_wqe_tls_progress_params_seg *params, u32 tis_tir_num)
+fill_progress_params(struct mlx5_wqe_tls_progress_params_seg *params, u32 tis_tir_num,
+		     u32 next_record_tcp_sn)
 {
 	u8 *ctx = params->ctx;
 
 	params->tis_tir_num = cpu_to_be32(tis_tir_num);
 
+	MLX5_SET(tls_progress_params, ctx, next_record_tcp_sn,
+		 next_record_tcp_sn);
 	MLX5_SET(tls_progress_params, ctx, record_tracker_state,
 		 MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
 	MLX5_SET(tls_progress_params, ctx, auth_state,
@@ -92,19 +97,22 @@ void
 mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
 				 u16 pc, u32 sqn,
 				 u32 tis_tir_num, bool fence,
+				 u32 next_record_tcp_sn,
 				 enum tls_offload_ctx_dir direction)
 {
 	struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+	u8 opmod = direction == TLS_OFFLOAD_CTX_DIR_TX ?
+		MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS :
+		MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS;
 
 #define PROGRESS_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
 
 	cseg->opmod_idx_opcode =
-		cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV |
-			    (MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS << 24));
+		cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV | (opmod << 24));
 	cseg->qpn_ds           = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
 					     PROGRESS_PARAMS_DS_CNT);
 	cseg->fm_ce_se         = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
 
-	fill_progress_params(&wqe->params, tis_tir_num);
+	fill_progress_params(&wqe->params, tis_tir_num, next_record_tcp_sn);
 }
 
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h +4 -0
index a325f34831176..7d7b63ae5c350 100644
@@ -21,6 +21,10 @@ u8 mlx5e_ktls_dumps_num_wqebbs(struct mlx5e_txqsq *sq, unsigned int nfrags,
 bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
 			      struct sk_buff *skb, int datalen,
 			      struct mlx5e_accel_tx_tls_state *state);
+void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+			      struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
+
+void mlx5e_ktls_handle_ctx_completion(struct mlx5e_icosq_wqe_info *wi);
 
 void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
 					   struct mlx5e_tx_wqe_info *wi,
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h +6 -0
index d1d747cb2dcb6..af8117d4bf884 100644
@@ -23,6 +23,11 @@ enum {
 int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
 		      struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);
 void mlx5e_ktls_del_tx(struct net_device *netdev, struct tls_context *tls_ctx);
+int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
+		      struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);
+void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx);
+int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
+			 u32 seq, u8 *rcd_sn);
 
 struct mlx5e_set_tls_static_params_wqe {
 	struct mlx5_wqe_ctrl_seg ctrl;
@@ -64,6 +69,7 @@ void
 mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
 				 u16 pc, u32 sqn,
 				 u32 tis_tir_num, bool fence,
+				 u32 next_record_tcp_sn,
 				 enum tls_offload_ctx_dir direction);
 
 #endif /* __MLX5E_TLS_UTILS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c +6 -0
index fba561ffe1d4e..dd9bbf3830564 100644
@@ -197,6 +197,7 @@ void mlx5e_tls_build_netdev(struct mlx5e_priv *priv)
 		return;
 	}
 
+	/* FPGA */
 	if (!mlx5_accel_is_tls_device(priv->mdev))
 		return;
 
@@ -226,6 +227,10 @@ int mlx5e_tls_init(struct mlx5e_priv *priv)
 	if (!tls)
 		return -ENOMEM;
 
+	tls->rx_wq = create_singlethread_workqueue("mlx5e_tls_rx");
+	if (!tls->rx_wq)
+		return -ENOMEM;
+
 	priv->tls = tls;
 	return 0;
 }
@@ -237,6 +242,7 @@ void mlx5e_tls_cleanup(struct mlx5e_priv *priv)
 	if (!tls)
 		return;
 
+	destroy_workqueue(tls->rx_wq);
 	kfree(tls);
 	priv->tls = NULL;
 }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h +1 -0
index 9015f3f7792d2..ca0c2ebb41a1c 100644
@@ -53,6 +53,7 @@ struct mlx5e_tls_sw_stats {
 
 struct mlx5e_tls {
 	struct mlx5e_tls_sw_stats sw_stats;
+	struct workqueue_struct *rx_wq;
 };
 
 struct mlx5e_tls_offload_context_tx {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c +13 -7
index 4d796fea906d1..182841322ce42 100644
@@ -278,9 +278,10 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
 	if (WARN_ON_ONCE(tls_ctx->netdev != netdev))
 		goto err_out;
 
-	if (MLX5_CAP_GEN(sq->channel->mdev, tls_tx))
+	if (mlx5_accel_is_ktls_tx(sq->channel->mdev))
 		return mlx5e_ktls_handle_tx_skb(tls_ctx, sq, skb, datalen, state);
 
+	/* FPGA */
 	skb_seq = ntohl(tcp_hdr(skb)->seq);
 	context = mlx5e_get_tls_tx_context(tls_ctx);
 	expected_seq = context->expected_seq;
@@ -354,12 +355,16 @@ static int tls_update_resync_sn(struct net_device *netdev,
 	return 0;
 }
 
-void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
-			     u32 *cqe_bcnt)
+void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+			     struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
 {
 	struct mlx5e_tls_metadata *mdata;
 	struct mlx5e_priv *priv;
 
+	if (likely(mlx5_accel_is_ktls_rx(rq->mdev)))
+		return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
+
+	/* FPGA */
 	if (!is_metadata_hdr_valid(skb))
 		return;
 
@@ -370,13 +375,13 @@ void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 		skb->decrypted = 1;
 		break;
 	case SYNDROM_RESYNC_REQUEST:
-		tls_update_resync_sn(netdev, skb, mdata);
-		priv = netdev_priv(netdev);
+		tls_update_resync_sn(rq->netdev, skb, mdata);
+		priv = netdev_priv(rq->netdev);
 		atomic64_inc(&priv->tls->sw_stats.rx_tls_resync_request);
 		break;
 	case SYNDROM_AUTH_FAILED:
 		/* Authentication failure will be observed and verified by kTLS */
-		priv = netdev_priv(netdev);
+		priv = netdev_priv(rq->netdev);
 		atomic64_inc(&priv->tls->sw_stats.rx_tls_auth_fail);
 		break;
 	default:
@@ -395,9 +400,10 @@ u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
 	if (!mlx5_accel_is_tls_device(mdev))
 		return 0;
 
-	if (MLX5_CAP_GEN(mdev, tls_tx))
+	if (mlx5_accel_is_ktls_device(mdev))
 		return mlx5e_ktls_get_stop_room(sq);
 
+	/* FPGA */
 	/* Resync SKB. */
 	return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
 }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h +2 -2
index 2a7b98531539f..8bb7906740425 100644
@@ -49,8 +49,8 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
 void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
 			     struct mlx5e_accel_tx_tls_state *state);
 
-void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
-			     u32 *cqe_bcnt);
+void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+			     struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
 
 #else
 
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +9 -0
index db18f602cdb2f..1b173d0aad1f4 100644
@@ -1441,6 +1441,7 @@ void mlx5e_close_icosq(struct mlx5e_icosq *sq)
 	struct mlx5e_channel *c = sq->channel;
 
 	mlx5e_destroy_sq(c->mdev, sq->sqn);
+	mlx5e_free_icosq_descs(sq);
 	mlx5e_free_icosq(sq);
 }
 
@@ -3851,6 +3852,7 @@ int mlx5e_set_features(struct net_device *netdev, netdev_features_t features)
 #ifdef CONFIG_MLX5_EN_ARFS
 	err |= MLX5E_HANDLE_FEATURE(NETIF_F_NTUPLE, set_feature_arfs);
 #endif
+	err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_TLS_RX, mlx5e_ktls_set_feature_rx);
 
 	if (err) {
 		netdev->features = oper_features;
@@ -5133,8 +5135,14 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_destroy_flow_steering;
 
+	err = mlx5e_accel_init_rx(priv);
+	if (err)
+		goto err_tc_nic_cleanup;
+
 	return 0;
 
+err_tc_nic_cleanup:
+	mlx5e_tc_nic_cleanup(priv);
 err_destroy_flow_steering:
 	mlx5e_destroy_flow_steering(priv);
 err_destroy_xsk_tirs:
@@ -5158,6 +5166,7 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
 
 static void mlx5e_cleanup_nic_rx(struct mlx5e_priv *priv)
 {
+	mlx5e_accel_cleanup_rx(priv);
 	mlx5e_tc_nic_cleanup(priv);
 	mlx5e_destroy_flow_steering(priv);
 	mlx5e_destroy_direct_tirs(priv, priv->xsk_tir);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +32 -1
index 6b3c82da199ce..d5cce529190d5 100644
@@ -578,6 +578,30 @@ bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq)
 	return !!err;
 }
 
+void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
+{
+	u16 sqcc;
+
+	sqcc = sq->cc;
+
+	while (sqcc != sq->pc) {
+		struct mlx5e_icosq_wqe_info *wi;
+		u16 ci;
+
+		ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc);
+		wi = &sq->db.wqe_info[ci];
+		sqcc += wi->num_wqebbs;
+#ifdef CONFIG_MLX5_EN_TLS
+		switch (wi->wqe_type) {
+		case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
+			mlx5e_ktls_handle_ctx_completion(wi);
+			break;
+		}
+#endif
+	}
+	sq->cc = sqcc;
+}
+
 int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 {
 	struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
@@ -633,6 +657,13 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 				break;
 			case MLX5E_ICOSQ_WQE_NOP:
 				break;
+#ifdef CONFIG_MLX5_EN_TLS
+			case MLX5E_ICOSQ_WQE_UMR_TLS:
+				break;
+			case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
+				mlx5e_ktls_handle_ctx_completion(wi);
+				break;
+#endif
 			default:
 				netdev_WARN_ONCE(cq->channel->netdev,
 						 "Bad WQE type in ICOSQ WQE info: 0x%x\n",
@@ -983,7 +1014,7 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
 	skb->mac_len = ETH_HLEN;
 
 #ifdef CONFIG_MLX5_EN_TLS
-	mlx5e_tls_handle_rx_skb(netdev, skb, &cqe_bcnt);
+	mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
 #endif
 
 	if (lro_num_seg > 1) {
drivers/net/ethernet/mellanox/mlx5/core/fw.c +2 -1
index a5fbe73435081..c3095863372cd 100644
@@ -35,6 +35,7 @@
 #include <linux/module.h>
 #include "mlx5_core.h"
 #include "../../mlxfw/mlxfw.h"
+#include "accel/tls.h"
 
 enum {
 	MCQS_IDENTIFIER_BOOT_IMG	= 0x1,
@@ -236,7 +237,7 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 			return err;
 	}
 
-	if (MLX5_CAP_GEN(dev, tls_tx)) {
+	if (mlx5_accel_is_ktls_tx(dev) || mlx5_accel_is_ktls_rx(dev)) {
 		err = mlx5_core_get_caps(dev, MLX5_CAP_TLS);
 		if (err)
 			return err;
Signature · 1 line
2.26.2
#9 lore raw

PATCH 9/11 net/mlx5e: kTLS, Add kTLS RX stats

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Add global and per-channel ethtool SW stats for the device offload.
Document the new counters in tls-offload.rst.

---
 Documentation/networking/tls-offload.rst      |  8 +++++++
 .../mellanox/mlx5/core/en_accel/ktls_rx.c     | 10 ++++++++
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 24 +++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/en_stats.h    | 15 ++++++++++++
 4 files changed, 57 insertions(+)
Documentation/networking/tls-offload.rst +8 -0
index f914e81fd3a64..44c4b19647746 100644
@@ -428,6 +428,14 @@ by the driver:
    which were part of a TLS stream.
  * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets
    which were successfully decrypted.
+ * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for
+   decryption.
+ * ``rx_tls_ooo`` - number of RX packets which were part of a TLS stream
+   but did not arrive in the expected order and triggered the resync procedure.
+ * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
+   (connection has finished).
+ * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
+   but were not decrypted due to unexpected error in the state machine.
  * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device
    for encryption of their TLS payload.
  * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +10 -0
index 83779cbc380a7..13e666403a5df 100644
@@ -6,6 +6,9 @@
 #include "en_accel/ktls_utils.h"
 #include "en_accel/fs_tcp.h"
 
+#define KTLS_STATS_INC(priv, rxq, fld) \
+		((priv)->channels.c[rxq]->rq.stats->fld++)
+
 struct accel_rule {
 	struct work_struct work;
 	struct mlx5e_priv *priv;
@@ -218,10 +221,14 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
 	switch (tls_offload) {
 	case CQE_TLS_OFFLOAD_DECRYPTED:
 		skb->decrypted = 1;
+		rq->stats->tls_decrypted_packets++;
+		rq->stats->tls_decrypted_bytes += *cqe_bcnt;
 		break;
 	case CQE_TLS_OFFLOAD_RESYNC:
+		rq->stats->tls_ooo++;
 		break;
 	default: /* CQE_TLS_OFFLOAD_ERROR: */
+		rq->stats->tls_err++;
 		break;
 	}
 }
@@ -280,6 +287,8 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
 	if (err)
 		goto err_post_wqes;
 
+	KTLS_STATS_INC(priv, rxq, tls_ctx);
+
 	return 0;
 
 err_post_wqes:
@@ -308,6 +317,7 @@ void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx)
 		 */
 		wait_for_completion(&priv_rx->add_ctx);
 
+	KTLS_STATS_INC(priv, priv_rx->rxq, tls_del);
 	if (priv_rx->rule.rule)
 		mlx5e_accel_fs_del_sk(priv_rx->rule.rule);
 
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +24 -0
index f009fe09e99b3..13d64e6142069 100644
@@ -163,6 +163,14 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_congst_umr) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_arfs_err) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_recover) },
+#ifdef CONFIG_MLX5_EN_TLS
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_decrypted_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_decrypted_bytes) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_ctx) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_del) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_ooo) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) },
+#endif
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_arm) },
@@ -275,6 +283,14 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
 		s->rx_congst_umr  += rq_stats->congst_umr;
 		s->rx_arfs_err    += rq_stats->arfs_err;
 		s->rx_recover     += rq_stats->recover;
+#ifdef CONFIG_MLX5_EN_TLS
+		s->rx_tls_decrypted_packets += rq_stats->tls_decrypted_packets;
+		s->rx_tls_decrypted_bytes   += rq_stats->tls_decrypted_bytes;
+		s->rx_tls_ctx               += rq_stats->tls_ctx;
+		s->rx_tls_del               += rq_stats->tls_del;
+		s->rx_tls_ooo               += rq_stats->tls_ooo;
+		s->rx_tls_err               += rq_stats->tls_err;
+#endif
 		s->ch_events      += ch_stats->events;
 		s->ch_poll        += ch_stats->poll;
 		s->ch_arm         += ch_stats->arm;
@@ -1475,6 +1491,14 @@ static const struct counter_desc rq_stats_desc[] = {
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, congst_umr) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, arfs_err) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, recover) },
+#ifdef CONFIG_MLX5_EN_TLS
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_decrypted_packets) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_decrypted_bytes) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_ctx) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_del) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_ooo) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) },
+#endif
 };
 
 static const struct counter_desc sq_stats_desc[] = {
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +15 -0
index 2b83ba9907140..b8d4488c9b328 100644
@@ -186,6 +186,13 @@ struct mlx5e_sw_stats {
 	u64 tx_tls_skip_no_sync_data;
 	u64 tx_tls_drop_no_sync_data;
 	u64 tx_tls_drop_bypass_req;
+
+	u64 rx_tls_decrypted_packets;
+	u64 rx_tls_decrypted_bytes;
+	u64 rx_tls_ctx;
+	u64 rx_tls_del;
+	u64 rx_tls_ooo;
+	u64 rx_tls_err;
 #endif
 
 	u64 rx_xsk_packets;
@@ -305,6 +312,14 @@ struct mlx5e_rq_stats {
 	u64 congst_umr;
 	u64 arfs_err;
 	u64 recover;
+#ifdef CONFIG_MLX5_EN_TLS
+	u64 tls_decrypted_packets;
+	u64 tls_decrypted_bytes;
+	u64 tls_ctx;
+	u64 tls_del;
+	u64 tls_ooo;
+	u64 tls_err;
+#endif
 };
 
 struct mlx5e_sq_stats {
Signature · 1 line
2.26.2
maintainer
#10 lore raw

Re: [net-next 09/11] net/mlx5e: kTLS, Add kTLS RX stats

Quoted 9 lines · On Fri, 29 May 2020 12:46:39 -0700 Saeed Mahameed wrote:
Documentation/networking/tls-offload.rst +1 -0
index f914e81fd3a64..44c4b19647746 100644
@@ -428,6 +428,14 @@ by the driver:
    which were part of a TLS stream.
  * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets
    which were successfully decrypted.
+ * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for

Documentation/networking/tls-offload.rst by the driver: · line 441

Quoted 10 lines
Documentation/networking/tls-offload.rst +7 -0
+   decryption.
+ * ``rx_tls_ooo`` - number of RX packets which were part of a TLS stream
+   but did not arrive in the expected order and triggered the resync procedure.
+ * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
+   (connection has finished).
+ * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
+   but were not decrypted due to unexpected error in the state machine.
  * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device
    for encryption of their TLS payload.
  * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets

Stack already has stats for some of these in /proc/net/tls_stat. Does this really need to be per device?

#11 lore raw

Re: [net-next 09/11] net/mlx5e: kTLS, Add kTLS RX stats

Quoted 25 lines · Jakub Kicinski
Quoted 24 lines · On Fri, 29 May 2020 12:46:39 -0700 Saeed Mahameed wrote:
Documentation/networking/tls-offload.rst +0 -0
index f914e81fd3a64..44c4b19647746 100644
@@ -428,6 +428,14 @@ by the driver:
    which were part of a TLS stream.
  * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX

packets
which were successfully decrypted.
+ * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for
+ decryption.
+ * ``rx_tls_ooo`` - number of RX packets which were part of a TLS stream
+ but did not arrive in the expected order and triggered the
resync procedure.
+ * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
+ (connection has finished).
+ * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
+ but were not decrypted due to unexpected error in the state
machine.

Quoted 8 lines
Quoted 5 lines

* ``tx_tls_encrypted_packets`` - number of TX packets passed to
the device
for encryption of their TLS payload.
* ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX
packets

Stack already has stats for some of these in /proc/net/tls_stat. Does this really need to be per device?

these are great for debug.. IMHO stats for offloaded flows per device are nice thing to have ..

#12 lore raw

PATCH 10/11 net/mlx5e: kTLS, Add kTLS RX resync support

In reply to Saeed Mahameed

From: Tariq Toukan <[email protected]>

Implement the RX resync procedure.

The HW offload of TLS decryption in RX side might get out-of-sync due to out-of-order reception of packets.
This requires SW intervention to update the HW context and get it back in-sync.

Performance:
CPU: Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz, 24 cores, HT off
NIC: ConnectX-6 Dx 100GbE dual port

Goodput (app-layer throughput) comparison:

+---------------+-------+-------+---------+
| # connections |   1   |   4   |    8    |
+---------------+-------+-------+---------+
| SW (Gbps)     |  7.26 | 24.70 |   50.30 |
+---------------+-------+-------+---------+
| HW (Gbps)     | 18.50 | 64.30 |   92.90 |
+---------------+-------+-------+---------+
| Speedup       | 2.55x | 2.56x | 1.85x * |
+---------------+-------+-------+---------+

* After linerate is reached, diff is observed in CPU util.

---
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   1 +
 .../mellanox/mlx5/core/en_accel/ktls_rx.c     | 322 +++++++++++++++++-
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     |   2 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c   |   7 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.h   |   2 +
 .../mellanox/mlx5/core/en_accel/ktls_utils.h  |  14 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |   6 +
 7 files changed, 347 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +1 -0
index ad1525aa5670b..934c18263de9a 100644
@@ -14,6 +14,7 @@ enum mlx5e_icosq_wqe_type {
 #ifdef CONFIG_MLX5_EN_TLS
 	MLX5E_ICOSQ_WQE_UMR_TLS,
 	MLX5E_ICOSQ_WQE_SET_PSV_TLS,
+	MLX5E_ICOSQ_WQE_GET_PSV_TLS,
 #endif
 };
 
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +320 -2

417 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +1 -1
index 9c34ffa55b328..0e6698d1b4ca9 100644
@@ -171,7 +171,7 @@ post_static_params(struct mlx5e_txqsq *sq,
 	pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
 	wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
 	mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_tx->crypto_info,
-				       priv_tx->tisn, priv_tx->key_id, fence,
+				       priv_tx->tisn, priv_tx->key_id, 0, fence,
 				       TLS_OFFLOAD_CTX_DIR_TX);
 	tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
 	sq->pc += num_wqebbs;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c +4 -3
index c1f1ad32ca4c2..ac29aeb8af492 100644
@@ -22,7 +22,7 @@ enum {
 static void
 fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
 		   struct tls12_crypto_info_aes_gcm_128 *info,
-		   u32 key_id)
+		   u32 key_id, u32 resync_tcp_sn)
 {
 	char *initial_rn, *gcm_iv;
 	u16 salt_sz, rec_seq_sz;
@@ -47,6 +47,7 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
 	MLX5_SET(tls_static_params, ctx, const_2, 2);
 	MLX5_SET(tls_static_params, ctx, encryption_standard,
 		 MLX5E_ENCRYPTION_STANDARD_TLS);
+	MLX5_SET(tls_static_params, ctx, resync_tcp_sn, resync_tcp_sn);
 	MLX5_SET(tls_static_params, ctx, dek_index, key_id);
 }
 
@@ -54,7 +55,7 @@ void
 mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
 			       u16 pc, u32 sqn,
 			       struct tls12_crypto_info_aes_gcm_128 *info,
-			       u32 tis_tir_num, u32 key_id,
+			       u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
 			       bool fence, enum tls_offload_ctx_dir direction)
 {
 	struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
@@ -74,7 +75,7 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
 	ucseg->flags = MLX5_UMR_INLINE;
 	ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
 
-	fill_static_params(&wqe->params, info, key_id);
+	fill_static_params(&wqe->params, info, key_id, resync_tcp_sn);
 }
 
 static void
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h +2 -0
index 7d7b63ae5c350..66e5063a22ce4 100644
@@ -25,6 +25,8 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
 			      struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
 
 void mlx5e_ktls_handle_ctx_completion(struct mlx5e_icosq_wqe_info *wi);
+void mlx5e_ktls_handle_get_psv_completion(struct mlx5e_icosq_wqe_info *wi,
+					  struct mlx5e_icosq *sq);
 
 void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
 					   struct mlx5e_tx_wqe_info *wi,
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h +13 -1
index af8117d4bf884..3dbc45e29828b 100644
@@ -41,12 +41,20 @@ struct mlx5e_set_tls_progress_params_wqe {
 	struct mlx5_wqe_tls_progress_params_seg params;
 };
 
+struct mlx5e_get_tls_progress_params_wqe {
+	struct mlx5_wqe_ctrl_seg ctrl;
+	struct mlx5_seg_get_psv  psv;
+};
+
 #define MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS \
 	(DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_static_params_wqe), MLX5_SEND_WQE_BB))
 
 #define MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS \
 	(DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
 
+#define MLX5E_KTLS_GET_PROGRESS_WQEBBS \
+	(DIV_ROUND_UP(sizeof(struct mlx5e_get_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
+
 #define MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi) \
 	((struct mlx5e_set_tls_static_params_wqe *)\
 	 mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_static_params_wqe)))
@@ -55,6 +63,10 @@ struct mlx5e_set_tls_progress_params_wqe {
 	((struct mlx5e_set_tls_progress_params_wqe *)\
 	 mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_progress_params_wqe)))
 
+#define MLX5E_TLS_FETCH_GET_PROGRESS_PARAMS_WQE(sq, pi) \
+	((struct mlx5e_get_tls_progress_params_wqe *)\
+	 mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_get_tls_progress_params_wqe)))
+
 #define MLX5E_TLS_FETCH_DUMP_WQE(sq, pi) \
 	((struct mlx5e_dump_wqe *)\
 	 mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe)))
@@ -63,7 +75,7 @@ void
 mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
 			       u16 pc, u32 sqn,
 			       struct tls12_crypto_info_aes_gcm_128 *info,
-			       u32 tis_tir_num, u32 key_id,
+			       u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
 			       bool fence, enum tls_offload_ctx_dir direction);
 void
 mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +6 -0
index d5cce529190d5..19bcd49224526 100644
@@ -596,6 +596,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
 		case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
 			mlx5e_ktls_handle_ctx_completion(wi);
 			break;
+		case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
+			mlx5e_ktls_handle_get_psv_completion(wi, sq);
+			break;
 		}
 #endif
 	}
@@ -663,6 +666,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 			case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
 				mlx5e_ktls_handle_ctx_completion(wi);
 				break;
+			case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
+				mlx5e_ktls_handle_get_psv_completion(wi, sq);
+				break;
 #endif
 			default:
 				netdev_WARN_ONCE(cq->channel->netdev,
Signature · 1 line
2.26.2
maintainer
#13 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 8 lines · On Fri, 29 May 2020 12:46:40 -0700 Saeed Mahameed wrote:

/* Re-sync */

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +7 -0
+static struct mlx5_wqe_ctrl_seg *
+resync_post_get_progress_params(struct mlx5e_icosq *sq,
+				struct mlx5e_ktls_offload_context_rx *priv_rx)
+{
+	struct mlx5e_ktls_rx_resync_ctx *resync = &priv_rx->resync;
+	struct mlx5e_get_tls_progress_params_wqe *wqe;
+	struct mlx5_wqe_ctrl_seg *cseg;

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c mlx5e_get_ktls_rx_priv_ctx(struct tls_context *tls_ctx) · line 247

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +10 -0
+	struct mlx5_seg_get_psv *psv;
+	u16 pi;
+
+	dma_sync_single_for_device(resync->priv->mdev->device,
+				   resync->dma_addr,
+				   PROGRESS_PARAMS_PADDED_SIZE,
+				   DMA_FROM_DEVICE);
+	BUILD_BUG_ON(MLX5E_KTLS_GET_PROGRESS_WQEBBS != 1);
+	if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)))
+		return ERR_PTR(-ENOSPC);

I thought you said that resync requests are guaranteed to never fail?

Quoted 23 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +23 -0
+	pi = mlx5e_icosq_get_next_pi(sq, 1);
+	wqe = MLX5E_TLS_FETCH_GET_PROGRESS_PARAMS_WQE(sq, pi);
+
+#define GET_PSV_DS_CNT (DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS))
+
+	cseg = &wqe->ctrl;
+	cseg->opmod_idx_opcode =
+		cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_GET_PSV |
+			    (MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS << 24));
+	cseg->qpn_ds =
+		cpu_to_be32((sq->sqn << MLX5_WQE_CTRL_QPN_SHIFT) | GET_PSV_DS_CNT);
+
+	psv = &wqe->psv;
+	psv->num_psv      = 1 << 4;
+	psv->l_key        = sq->channel->mkey_be;
+	psv->psv_index[0] = cpu_to_be32(priv_rx->tirn);
+	psv->va           = cpu_to_be64(priv_rx->resync.dma_addr);
+
+	icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_GET_PSV_TLS, 1, priv_rx);
+	sq->pc++;
+
+	return cseg;
+}
#14 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Fri, 29 May 2020 12:46:40 -0700 Saeed Mahameed wrote:

/* Re-sync */

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +3 -0
+static struct mlx5_wqe_ctrl_seg *
+resync_post_get_progress_params(struct mlx5e_icosq *sq,
+				struct mlx5e_ktls_offload_context_rx

*priv_rx)

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +9 -0
+{
+	struct mlx5e_ktls_rx_resync_ctx *resync = &priv_rx->resync;
+	struct mlx5e_get_tls_progress_params_wqe *wqe;
+	struct mlx5_wqe_ctrl_seg *cseg;
+	struct mlx5_seg_get_psv *psv;
+	u16 pi;
+
+	dma_sync_single_for_device(resync->priv->mdev->device,
+				   resync->dma_addr,
Quoted 9 lines

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c mlx5e_get_ktls_rx_priv_ctx(struct tls_context *tls_ctx) · line 247

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +4 -0
+				   PROGRESS_PARAMS_PADDED_SIZE,
+				   DMA_FROM_DEVICE);
+	BUILD_BUG_ON(MLX5E_KTLS_GET_PROGRESS_WQEBBS != 1);
+	if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc,

1)))

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +1 -0
+		return ERR_PTR(-ENOSPC);

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review, but basically with the current mlx5 arch, it is impossible to guarantee this unless we open 1 service queue per ktls offloads and that is going to be an overkill!

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

Quoted 26 lines
Quoted 26 lines
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +4 -0
+	pi = mlx5e_icosq_get_next_pi(sq, 1);
+	wqe = MLX5E_TLS_FETCH_GET_PROGRESS_PARAMS_WQE(sq, pi);
+
+#define GET_PSV_DS_CNT (DIV_ROUND_UP(sizeof(*wqe),

MLX5_SEND_WQE_DS))

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +5 -0
+
+	cseg = &wqe->ctrl;
+	cseg->opmod_idx_opcode =
+		cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_GET_PSV |
+			    (MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS <<

24));

(quoted diff) +2 -0
+	cseg->qpn_ds =
+		cpu_to_be32((sq->sqn << MLX5_WQE_CTRL_QPN_SHIFT) |

GET_PSV_DS_CNT);

drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +12 -0
+
+	psv = &wqe->psv;
+	psv->num_psv      = 1 << 4;
+	psv->l_key        = sq->channel->mkey_be;
+	psv->psv_index[0] = cpu_to_be32(priv_rx->tirn);
+	psv->va           = cpu_to_be64(priv_rx->resync.dma_addr);
+
+	icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_GET_PSV_TLS, 1, priv_rx);
+	sq->pc++;
+
+	return cseg;
+}
maintainer
#15 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 3 lines · On Fri, 29 May 2020 20:44:29 +0000 Saeed Mahameed wrote:
Quoted 1 line

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review,

Boris ;)

Quoted 3 lines

but basically with the current mlx5 arch, it is impossible to guarantee this unless we open 1 service queue per ktls offloads and that is going to be an overkill!

IIUC every ooo packet causes a resync request in your implementation - is that true?

It'd be great to have more information about the operation of the device in the commit message..

Quoted 2 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

We don't want to add extra load on every record if HW offload is enabled. That's why the next record hint backs off, checks socket state etc.

BTW I also don't understand why mlx5e_ktls_rx_resync() has a tls_offload_rx_force_resync_request(sk) at the end. If the update from the NIC comes with a later seq than current, request the sync for _that_ seq. I don't understand the need to force a call back on every record here.

Also if the sync failed because queue was full, I don't see how forcing another sync attempt for the next record is going to match?

#16 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 14 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 29 May 2020 20:44:29 +0000 Saeed Mahameed wrote:
Quoted 2 lines

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review,

Boris ;)

Quoted 5 lines

but basically with the current mlx5 arch, it is impossible to guarantee
this unless we open 1 service queue per ktls offloads and that is going
to be an overkill!

Quoted 4 lines

IIUC every ooo packet causes a resync request in your implementation -
is that true?

For tx yes, for RX i am not sure, this is a hw flow that I am not fully familiar with.

Anyway according to Tariq, The hw might generate more than one resync request on the same flow, and this is all being handled by the driver correctly. I am not sure if this is what you are looking for.

Maybe Tariq/Boris can elaborate more on the hw resync mechanism.

Quoted 3 lines

It'd be great to have more information about the operation of the device in the commit message..

How about:

Resync flow occurs when packets have been lost and the device lost track of TLS records. The device attempts to resync by tracking TLS records, and sends a resync request to driver. The TLS Progress Params Context holds the TCP-SN of the record where the device began tracking records and counting them. The driver will acknowledge the TCP-SN if it matches a legal record by setting the TLS Static Params Context.

?
we can elaborate more with a step by step procedure.. if you think it is required.

Quoted 16 lines
Quoted 4 lines

This is a rare corner case anyway, where more than 1k tcp connections
sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity,
not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

We don't want to add extra load on every record if HW offload is enabled. That's why the next record hint backs off, checks socket state etc.

Quoted 5 lines

BTW I also don't understand why mlx5e_ktls_rx_resync() has a tls_offload_rx_force_resync_request(sk) at the end. If the update from the NIC comes with a later seq than current, request the sync for _that_ seq. I don't understand the need to force a call back on every record here.

Good point theoretically should work, unless we have some limitations that i am not seeing, i will let Tariq comment on this.

Quoted 4 lines

Also if the sync failed because queue was full, I don't see how forcing
another sync attempt for the next record is going to match?

In this case i guess we need to abort and wait for the hw to issue anew resync request ..

#17 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 47 lines · Saeed Mahameed
Quoted 14 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 29 May 2020 20:44:29 +0000 Saeed Mahameed wrote:
Quoted 2 lines

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review,

Boris ;)

Quoted 5 lines

but basically with the current mlx5 arch, it is impossible to guarantee
this unless we open 1 service queue per ktls offloads and that is going
to be an overkill!

Quoted 5 lines

IIUC every ooo packet causes a resync request in your implementation
-
is that true?

For tx yes, for RX i am not sure, this is a hw flow that I am not fully
familiar with.

Anyway according to Tariq, The hw might generate more than one resync request on the same flow, and this is all being handled by the driver correctly. I am not sure if this is what you are looking for.

Maybe Tariq/Boris can elaborate more on the hw resync mechanism.

Quoted 3 lines

It'd be great to have more information about the operation of the device in the commit message..

How about:

Resync flow occurs when packets have been lost and the device lost track of TLS records. The device attempts to resync by tracking TLS records, and sends a resync request to driver. The TLS Progress Params
Context holds the TCP-SN of the record where the device began tracking
records and counting them. The driver will acknowledge the TCP-SN if it
matches a legal record by setting the TLS Static Params Context.

Quoted 21 lines

?
we can elaborate more with a step by step procedure.. if you think it is required.

Quoted 17 lines
Quoted 4 lines

This is a rare corner case anyway, where more than 1k tcp connections
sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity,
not efficiency under steady load. So it matters a lot how the system
behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

We don't want to add extra load on every record if HW offload is enabled. That's why the next record hint backs off, checks socket state etc.

What we can do here is instead of failing when the queue is full, a resync request will keep trying and exponentially backoff
up to once per second. so eventually the system will not overload if the hw queue can't keep up, and eventually the latest hw resync request will be handled.

Quoted 9 lines
Quoted 5 lines

BTW I also don't understand why mlx5e_ktls_rx_resync() has a tls_offload_rx_force_resync_request(sk) at the end. If the update from the NIC comes with a later seq than current, request the sync for _that_ seq. I don't understand the need to force a call back on every record here.

Good point theoretically should work, unless we have some limitations that i am not seeing, i will let Tariq comment on this.

I think same as above, we can hint to the hw _that_ new seq, and will backoff until the hw catches up with sw and issues a new valid resync request.

Quoted 6 lines
Quoted 3 lines

Also if the sync failed because queue was full, I don't see how forcing
another sync attempt for the next record is going to match?

In this case i guess we need to abort and wait for the hw to issue anew resync request ..

#18 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

In reply to Jakub Kicinski
Quoted 7 lines · Jakub Kicinski
Quoted 3 lines · On Fri, 29 May 2020 20:44:29 +0000 Saeed Mahameed wrote:
Quoted 1 line

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review,

Boris ;)

I didn't say we are perfect, just that we can make a trade-off here, and currently this is the simplest version that our team came up with for this series. As a first step, I think it is reasonable. But, I expect that we will improve it in the future.

Quoted 3 lines
Quoted 3 lines

but basically with the current mlx5 arch, it is impossible to guarantee this unless we open 1 service queue per ktls offloads and that is going to be an overkill!

I disagree, there are many ways to guarantee reliability here. For example, we can sleep/spin until there is space in the queue or rely on work stealing to let a later operation execute this one.

Quoted 4 lines

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

Quoted 3 lines

It'd be great to have more information about the operation of the device in the commit message..

I'll try to clarify the resync flow here.
As always, the packet that requires resync is marked as such in the CQE.
However, unlike previous devices, the TCP sequence (tcpsn) where the HW found a header is not provided in the CQE.
Instead, tcpsn is queried from HW asynchronously by the driver.
We employ the force resync approach to guarantee that we can log all resync locations between the received packet and the HW query response.
We check the asynchronous HW query response against all resync values between the packet that triggered the resync and now. If one of them matches, then resync can be completed immediately. Otherwise, the driver keeps waiting for the correct resync.

Quoted 8 lines
Quoted 2 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

I agree that this is not that rare, and it may be improved both in future patches and hardware. Do you think it is critical to improve it now, and not in a follow-up series?

Quoted 11 lines

We don't want to add extra load on every record if HW offload is enabled. That's why the next record hint backs off, checks socket state etc.

BTW I also don't understand why mlx5e_ktls_rx_resync() has a tls_offload_rx_force_resync_request(sk) at the end. If the update from the NIC comes with a later seq than current, request the sync for _that_ seq. I don't understand the need to force a call back on every record here.

The extra load here is minimal, i.e. a single call from TLS to the driver, which usually just logs the information.

Quoted 3 lines

Also if the sync failed because queue was full, I don't see how forcing another sync attempt for the next record is going to match?

It doesn't, and if sync failed then we should stop trying to force resync.

maintainer
#19 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 22 lines · On Sun, 31 May 2020 15:06:28 +0300 Boris Pismenny wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 3 lines · On Fri, 29 May 2020 20:44:29 +0000 Saeed Mahameed wrote:
Quoted 1 line

I thought you said that resync requests are guaranteed to never fail?

I didn't say that :), maybe tariq did say this before my review,

Boris ;)

I didn't say we are perfect, just that we can make a trade-off here, and currently this is the simplest version that our team came up with for this series. As a first step, I think it is reasonable. But, I expect that we will improve it in the future.

Quoted 3 lines
Quoted 3 lines

but basically with the current mlx5 arch, it is impossible to guarantee this unless we open 1 service queue per ktls offloads and that is going to be an overkill!

I disagree, there are many ways to guarantee reliability here. For example, we can sleep/spin until there is space in the queue or rely on work stealing to let a later operation execute this one.

Quoted 6 lines
Quoted 3 lines

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

I mean until device is back in sync every frame kicks off resync_update_sn() and tries to queue the work, right?

Quoted 4 lines
Quoted 3 lines

It'd be great to have more information about the operation of the device in the commit message..

Quoted 11 lines

I'll try to clarify the resync flow here.
As always, the packet that requires resync is marked as such in the CQE. However, unlike previous devices, the TCP sequence (tcpsn) where the HW found a header is not provided in the CQE. Instead, tcpsn is queried from HW asynchronously by the driver. We employ the force resync approach to guarantee that we can log all resync locations between the received packet and the HW query response. We check the asynchronous HW query response against all resync values between the packet that triggered the resync and now. If one of them matches, then resync can be completed immediately. Otherwise, the driver keeps waiting for the correct resync.

Thanks, makes sense.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

Quoted 6 lines
Quoted 2 lines

overload -> drops -> TLS steams go out of sync -> all try to resync

I agree that this is not that rare, and it may be improved both in future patches and hardware. Do you think it is critical to improve it now, and not in a follow-up series?

It's not a blocker for me, although if this makes it into 5.8 there will not be a chance to improve before net-next closes, so depends if you want to risk it and support the code as is...

Quoted 11 lines
Quoted 10 lines

We don't want to add extra load on every record if HW offload is enabled. That's why the next record hint backs off, checks socket state etc.

BTW I also don't understand why mlx5e_ktls_rx_resync() has a tls_offload_rx_force_resync_request(sk) at the end. If the update from the NIC comes with a later seq than current, request the sync for _that_ seq. I don't understand the need to force a call back on every record here.

Quoted 2 lines

The extra load here is minimal, i.e. a single call from TLS to the driver, which usually just logs the information.

Oh yes, this one is not about extra load, I just don't know what that code is trying to achieve.

Quoted 6 lines
Quoted 3 lines

Also if the sync failed because queue was full, I don't see how forcing another sync attempt for the next record is going to match?

It doesn't, and if sync failed then we should stop trying to force resync.

#20 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 11 lines · Jakub Kicinski
Quoted 7 lines · On Sun, 31 May 2020 15:06:28 +0300 Boris Pismenny wrote:
Quoted 3 lines · Jakub Kicinski

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

I mean until device is back in sync every frame kicks off resync_update_sn() and tries to queue the work, right?

Nope, only the first frame triggers resync_update_sn, so as to keep the process efficient and avoid spamming the system with resync requests. Per-flow, the device will try again to trigger resync_update_sn only if it gets out of sync due to out-of-sequence record headers.

maintainer
#21 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 9 lines · On Tue, 2 Jun 2020 07:23:53 +0300 Boris Pismenny wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Sun, 31 May 2020 15:06:28 +0300 Boris Pismenny wrote:
Quoted 3 lines · Jakub Kicinski

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

Quoted 8 lines
Quoted 3 lines

I mean until device is back in sync every frame kicks off resync_update_sn() and tries to queue the work, right?

Nope, only the first frame triggers resync_update_sn, so as to keep the process efficient and avoid spamming the system with resync requests. Per-flow, the device will try again to trigger resync_update_sn only if it gets out of sync due to out-of-sequence record headers.

It'd be good to clarify what the ooo counter counts in the documentation, it sounds like it counts first TLS header HW found after seq discontinuity is detected?

In fact calling this a ooo counter may be slightly misleading, I like the nfp counters much more: tx_tls_resync_req_ok and tx_tls_resync_req_ign.

#22 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 19 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 2 Jun 2020 07:23:53 +0300 Boris Pismenny wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Sun, 31 May 2020 15:06:28 +0300 Boris Pismenny wrote:
Quoted 3 lines · Jakub Kicinski

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

Quoted 8 lines
Quoted 3 lines

I mean until device is back in sync every frame kicks off resync_update_sn() and tries to queue the work, right?

Nope, only the first frame triggers resync_update_sn, so as to keep the process efficient and avoid spamming the system with resync requests. Per-flow, the device will try again to trigger resync_update_sn only if it gets out of sync due to out-of-sequence record headers.

Quoted 4 lines

It'd be good to clarify what the ooo counter counts in the documentation, it sounds like it counts first TLS header HW found after seq discontinuity is detected?

Sure NP, I will add it.
It counts the number of times the device marked the tls resync bit in a packet's completion, this happens when hw is tracking (but not offloading) and found a tls magic.

Quoted 4 lines

In fact calling this a ooo counter may be slightly misleading, I like the nfp counters much more: tx_tls_resync_req_ok and tx_tls_resync_req_ign.

I can rename. NP.

#23 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

In reply to Jakub Kicinski
Quoted 18 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 2 Jun 2020 07:23:53 +0300 Boris Pismenny wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Sun, 31 May 2020 15:06:28 +0300 Boris Pismenny wrote:
Quoted 3 lines · Jakub Kicinski

IIUC every ooo packet causes a resync request in your implementation - is that true?

No, only header loss. We never required a resync per OOO packet. I'm not sure why would you think that.

Quoted 8 lines
Quoted 3 lines

I mean until device is back in sync every frame kicks off resync_update_sn() and tries to queue the work, right?

Nope, only the first frame triggers resync_update_sn, so as to keep the process efficient and avoid spamming the system with resync requests. Per-flow, the device will try again to trigger resync_update_sn only if it gets out of sync due to out-of-sequence record headers.

Quoted 7 lines

It'd be good to clarify what the ooo counter counts in the documentation, it sounds like it counts first TLS header HW found after seq discontinuity is detected?

In fact calling this a ooo counter may be slightly misleading, I like the nfp counters much more: tx_tls_resync_req_ok and tx_tls_resync_req_ign.

I agree. We will add it to mlx5 and document.Thanks!

#24 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

In reply to Jakub Kicinski
Quoted 12 lines · Jakub Kicinski
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

Quoted 8 lines
Quoted 3 lines

I agree that this is not that rare, and it may be improved both in future patches and hardware. Do you think it is critical to improve it now, and not in a follow-up series?

It's not a blocker for me, although if this makes it into 5.8 there will not be a chance to improve before net-next closes, so depends if you want to risk it and support the code as is...

Hi Jakub,
Thanks for your comments.

This is just the beginning of this driver's offload support. I will continue working on enhancements and improvements in next kernels. We have several enhancements in plans.

For now, if no real blockers, I think it's in a good shape to start with and make it to the kernel.

IMHO, this specific issue of better handling the resync failure in driver can be addressed in stages:

1. As a fix: stop asking the stack for resync re-calls. If a resync attempt fails, terminate any resync attempts for the specific connection. If there's room for a re-spin I can provide today. Otherwise it is a simple fix that can be addressed in the early rc's in -net. What do you think?

2. Recover: this is an enhancement to be done in future kernels, where the driver internally and independently recovers from failed attempts and makes sure the are processed when there's enough room on the SQ again. Without the stack being engaged.

Thanks,
Tariq

maintainer
#25 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 41 lines · On Tue, 2 Jun 2020 14:32:44 +0300 Tariq Toukan wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

Quoted 8 lines
Quoted 3 lines

I agree that this is not that rare, and it may be improved both in future patches and hardware. Do you think it is critical to improve it now, and not in a follow-up series?

It's not a blocker for me, although if this makes it into 5.8 there will not be a chance to improve before net-next closes, so depends if you want to risk it and support the code as is...

Hi Jakub,
Thanks for your comments.

This is just the beginning of this driver's offload support. I will continue working on enhancements and improvements in next kernels. We have several enhancements in plans.

For now, if no real blockers, I think it's in a good shape to start with and make it to the kernel.

IMHO, this specific issue of better handling the resync failure in driver can be addressed in stages:

1. As a fix: stop asking the stack for resync re-calls. If a resync attempt fails, terminate any resync attempts for the specific connection. If there's room for a re-spin I can provide today. Otherwise it is a simple fix that can be addressed in the early rc's in -net. What do you think?

Quoted 4 lines

2. Recover: this is an enhancement to be done in future kernels, where the driver internally and independently recovers from failed attempts and makes sure the are processed when there's enough room on the SQ again. Without the stack being engaged.

IIUC the HW asks for a resync at the first record after a specific seq (the record header is in the frame that carried the OOO marking, right?)

Can we make the core understand those semantics and avoid trying to resync at the wrong record?

#26 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 47 lines · Jakub Kicinski
Quoted 41 lines · On Tue, 2 Jun 2020 14:32:44 +0300 Tariq Toukan wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

This is a rare corner case anyway, where more than 1k tcp connections sharing the same RX ring will request resync at the same exact moment.

IDK about that. Certain applications are architected for max capacity, not efficiency under steady load. So it matters a lot how the system behaves under stress. What if this is the chain of events:

overload -> drops -> TLS steams go out of sync -> all try to resync

Quoted 8 lines
Quoted 3 lines

I agree that this is not that rare, and it may be improved both in future patches and hardware. Do you think it is critical to improve it now, and not in a follow-up series?

It's not a blocker for me, although if this makes it into 5.8 there will not be a chance to improve before net-next closes, so depends if you want to risk it and support the code as is...

Hi Jakub,
Thanks for your comments.

This is just the beginning of this driver's offload support. I will continue working on enhancements and improvements in next kernels. We have several enhancements in plans.

For now, if no real blockers, I think it's in a good shape to start with and make it to the kernel.

IMHO, this specific issue of better handling the resync failure in driver can be addressed in stages:

1. As a fix: stop asking the stack for resync re-calls. If a resync attempt fails, terminate any resync attempts for the specific connection. If there's room for a re-spin I can provide today. Otherwise it is a simple fix that can be addressed in the early rc's in -net. What do you think?

Quoted 4 lines

2. Recover: this is an enhancement to be done in future kernels, where the driver internally and independently recovers from failed attempts and makes sure the are processed when there's enough room on the SQ again. Without the stack being engaged.

Quoted 6 lines

IIUC the HW asks for a resync at the first record after a specific seq (the record header is in the frame that carried the OOO marking, right?)

Can we make the core understand those semantics and avoid trying to resync at the wrong record?

HW asks for a resync when it is in tracking mode and identifies the magic, so it calculates the expected seq of next record.
This seq is not part of the completion (for now, this is a planned enhancement), so the device driver posts a request to the device to get the seq, and then the driver hopefully approve it (by another post to the HW) after comparing it to the stack sw seq.

As long as the device driver does not know the HW expected seq, it cannot provide a seq to the stack. So force resync is used.

We can think of an optimization here, it is doable.

maintainer
#27 lore raw

Re: [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support

Quoted 13 lines · On Wed, 3 Jun 2020 10:02:33 +0300 Tariq Toukan wrote:
Quoted 5 lines

IIUC the HW asks for a resync at the first record after a specific seq (the record header is in the frame that carried the OOO marking, right?)

Can we make the core understand those semantics and avoid trying to resync at the wrong record?

HW asks for a resync when it is in tracking mode and identifies the magic, so it calculates the expected seq of next record. This seq is not part of the completion (for now, this is a planned enhancement), so the device driver posts a request to the device to get the seq, and then the driver hopefully approve it (by another post to the HW) after comparing it to the stack sw seq.

Quoted 2 lines

As long as the device driver does not know the HW expected seq, it cannot provide a seq to the stack. So force resync is used.

Right, what I was trying to say is that the HW will likely latch on the first magic in the TCP segment, so maybe the driver and the core can reasonably expect that. Driver can tell the core to provide first record after TCP seq_no X. Otherwise if the TCP socket is backed up driver may get a very old record.

Just clarifying what I was trying to say, not sure how that fits your device.

Quoted 1 line

We can think of an optimization here, it is doable.

#28 lore raw

PATCH 11/11 net/mlx5e: kTLS, Improve rx handler function call

In reply to Saeed Mahameed

Prior to this patch mlx5e tls rx handler was called unconditionally on all rx packets and the decision whether a packet is a valid tls packet is done inside that function. A function call can be expensive especially for regular rx packet rate. To avoid this, check the tls validity before jumping into the tls rx handler.

While at it, split between kTLS device offload rx handler and FPGA tls rx handler using a similar method.

---
 .../mellanox/mlx5/core/en_accel/ktls_rx.c     |  7 +----
 .../mellanox/mlx5/core/en_accel/tls_rxtx.c    | 12 +++------
 .../mellanox/mlx5/core/en_accel/tls_rxtx.h    | 26 +++++++++++++++++--
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  5 ++--
 4 files changed, 30 insertions(+), 20 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +1 -6
index 703ce78d54043..b0e7eb92d7174 100644
@@ -481,12 +481,7 @@ int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
 void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
 			      struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
 {
-	u8 tls_offload = get_cqe_tls_offload(cqe);
-
-	if (likely(tls_offload == CQE_TLS_OFFLOAD_NOT_DECRYPTED))
-		return;
-
-	switch (tls_offload) {
+	switch (get_cqe_tls_offload(cqe)) {
 	case CQE_TLS_OFFLOAD_DECRYPTED:
 		skb->decrypted = 1;
 		rq->stats->tls_decrypted_packets++;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c +3 -9
index 182841322ce42..b0c31d49ff8db 100644
@@ -355,19 +355,13 @@ static int tls_update_resync_sn(struct net_device *netdev,
 	return 0;
 }
 
-void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
-			     struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
+/* FPGA tls rx handler */
+void mlx5e_tls_handle_rx_skb_metadata(struct mlx5e_rq *rq, struct sk_buff *skb,
+				      u32 *cqe_bcnt)
 {
 	struct mlx5e_tls_metadata *mdata;
 	struct mlx5e_priv *priv;
 
-	if (likely(mlx5_accel_is_ktls_rx(rq->mdev)))
-		return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
-
-	/* FPGA */
-	if (!is_metadata_hdr_valid(skb))
-		return;
-
 	/* Use the metadata */
 	mdata = (struct mlx5e_tls_metadata *)(skb->data + ETH_HLEN);
 	switch (mdata->content.recv.syndrome) {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h +24 -2
index 8bb7906740425..08cb0f91fcd09 100644
@@ -34,6 +34,7 @@
 #ifndef __MLX5E_TLS_RXTX_H__
 #define __MLX5E_TLS_RXTX_H__
 
+#include "accel/accel.h"
 #include "en_accel/ktls_txrx.h"
 
 #ifdef CONFIG_MLX5_EN_TLS
@@ -49,11 +50,32 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
 void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
 			     struct mlx5e_accel_tx_tls_state *state);
 
-void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
-			     struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
+void mlx5e_tls_handle_rx_skb_metadata(struct mlx5e_rq *rq, struct sk_buff *skb,
+				      u32 *cqe_bcnt);
+
+static inline bool mlx5e_accel_is_tls(struct mlx5_cqe64 *cqe, struct sk_buff *skb)
+{
+	return get_cqe_tls_offload(cqe) || is_metadata_hdr_valid(skb);
+}
+
+static inline void
+mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+			struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
+{
+	if (likely(mlx5e_accel_is_tls(cqe, skb)))
+		return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
+
+	/* FPGA */
+	return mlx5e_tls_handle_rx_skb_metadata(rq, skb, cqe_bcnt);
+}
 
 #else
 
+static inline bool
+mlx5e_accel_is_tls(struct mlx5_cqe64 *cqe, struct sk_buff *skb) { return false; }
+static inline void
+mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+			struct mlx5_cqe64 *cqe, u32 *cqe_bcnt) {}
 static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
 {
 	return 0;
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +2 -3
index 19bcd49224526..35ff190daaa1e 100644
@@ -1019,9 +1019,8 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
 
 	skb->mac_len = ETH_HLEN;
 
-#ifdef CONFIG_MLX5_EN_TLS
-	mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
-#endif
+	if (unlikely(mlx5e_accel_is_tls(cqe, skb)))
+		mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
 
 	if (lro_num_seg > 1) {
 		mlx5e_lro_update_hdr(skb, cqe, cqe_bcnt);
Signature · 1 line
2.26.2