Skip to contentSkip to conversation

nvme-tcp receive and tarnsmit offloads

PATCH In review Boris Pismenny 8 participants 62 messages Jul 22, 2021 – Aug 10, 2021 1 message not in the archive

Subsystems (14)
lore
Share & tools
lore

Boris Pismenny

author reviewer
#0 lore raw

v5 cover nvme-tcp receive and tarnsmit offloads

From: Boris Pismenny <[email protected]>

Changes since v4:
=========================================
* Add transmit offload patches
* Use one feature bit for both receive and transmit offload

Changes since v3:
=========================================
* Use DDP_TCP ifdefs in iov_iter and skb iterators to minimize impact when compiled out (Christoph)
* Simplify netdev references and reduce the use of get_netdev_for_sock (Sagi)
* Avoid "static" in it's own line, move it one line down (Christoph)
* Pass (queue, skb, *offset) and retrieve the pdu_seq in nvme_tcp_resync_response (Sagi)
* Add missing assignment of offloading_netdev to null in offload_limits error case (Sagi)
* Set req->offloaded = false once -- the lifetime rules are:
set to false on cmd_setup / set to true when ddp setup succeeds (Sagi)
* Replace pr_info_ratelimited with dev_info_ratelimited (Sagi)
* Add nvme_tcp_complete_request and invoke it from two similar call sites (Sagi)
* Introduce nvme_tcp_req_map_sg earlier in the series (Sagi)
* Add nvme_tcp_consume_skb and put into it a hunk from nvme_tcp_recv_data to handle copy with and without offload

Changes since v2:
=========================================
* Use skb->ddp_crc for copy offload to avoid skb_condense
* Default mellanox driver support to no (experimental feature)
* In iov_iter use non-ddp functions for kvec and iovec
* Remove typecasting in nvme-tcp

Changes since v1:
=========================================
* Rework iov_iter copy skip if src==dst to be less intrusive (David Ahern)
* Add tcp-ddp documentation (David Ahern)
* Refactor mellanox driver patches into more patches (Saeed Mahameed)
* Avoid pointer casting (David Ahern)
* Rename nvme-tcp offload flags (Shai Malin)
* Update cover-letter according to the above

Changes since RFC v1:
=========================================
* Split mlx5 driver patches to several commits
* Fix nvme-tcp handling of recovery flows. In particular, move queue offlaod init/teardown to the start/stop functions.

# Overview
=========================================
This series adds support for nvme-tcp receive and transmit offloads which do not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers
2. CRC calculation and verification for received PDU

The series implements these as a generic offload infrastructure for storage protocols, which we call TCP Direct Data Placement (TCP_DDP) and TCP DDP CRC, respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC. Future implementations can reuse the same infrastructure for other protcols such as iSCSI.

Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html

# Initialization and teardown:
=========================================
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by calling nvme_tcp_offload_socket with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usually with no offload. If it succeeds then nvme_tcp_setup_ddp and nvme_tcp_teardown_ddp may be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of nvme-tcp in any way besides adding the option to call the above mentioned NDO operations.

For the admin queue, nvme-tcp does not initialize the offload. Instead, nvme-tcp calls the driver to configure limits for the controller, such as max_hw_sectors and max_segments; these must be limited to accomodate potential HW resource limits, and to improve performance.

If some error occured, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.

# Copy offload works as follows:
=========================================
The nvme-tcp layer calls the NIC drive to map block layer buffers to ccid using nvme_tcp_setup_ddp before sending the read request. When the repsonse is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer; this SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once nvme-tcp attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function (memcpy_to_page):
if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the nvme-tcp layer releases the NIC HW context be calling nvme_tcp_teardown_ddp which asynchronously unmaps the buffers from NIC HW.

As the copy skip change is in a sensative function, we are careful to avoid changing it. To that end, we create alternative skb copy and hash iterators that skip copy/hash if (src == dst). Nvme-tcp is the first user for these.

# Asynchronous completion:
=========================================
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance criticial, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).

An alternative approach is to move all the functions related to coping from SKBs to the block layer buffers inside the nvme-tcp code - about 200 LOC.

# CRC receive offload works as follows:
=========================================
After offload is initialized, we use the SKB's ddp_crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (ddp_crc != 1), then software must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.

# CRC transmit offload works as follows:
=========================================
The sending layer (e.g., nvme-tcp) sets the MSG_DDP_CRC when sending messages down to TCP. Thereafter TCP will mark corresponding SKBs with the ddp_crc bit. This ensures that CRC offload takes place for this packets, similarly to how skb->decrtypted is used for TLS.

Additionally, nvme-tcp maintains a mapping between TCP sequence numbers and PDU data which allows the driver to handle reordered/retransmitted packet offload by resynchronizing the device's CRC state.

# SKB changes:
=========================================
The CRC offload requires an additional bit in the SKB, which is useful for preventing the coalescing of SKB with different crc offload values. This bit is similar in concept to the "decrypted" bit.

# Performance:
=========================================
The expected performance gain from this offload varies with the block size. We perform a CPU cycles breakdown of the copy/CRC operations in nvme-tcp fio random read workloads:
For 4K blocks we see up to 11% improvement for a 100% read fio workload, while for 128K blocks we see upto 52%. If we run nvme-tcp, and skip these operations, then we observe a gain of about 1.1x and 2x respectively.

# Resynchronization:
=========================================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the driver, regarding a possible location of a PDU header. Followed by a response from the nvme-tcp driver.

This flow is rare, and it should happen only after packet loss or reordering events that involve nvme-tcp PDU headers.

# The patches are organized as follows:
=========================================
Patches 1,3 the infrastructure for all TCP DDP.
and TCP DDP CRC offloads, respectively.

Patch 2         the iov_iter change to skip copy if (src == dst).
Patch 4         exposes the get_netdev_for_sock function from TLS.
Patch 5         NVMe-TCP changes to call NIC driver on queue init/teardown.
Patches 6       NVMe-TCP changes to call NIC driver on IO operation.

setup/teardown, and support async completions.
Patches 7 NVMe-TCP changes to support CRC offload on receive.
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patches 8 NVMe-TCP handling of netdev events: stop the offload if
netdev is going down.
Patches 9-19 implement support for NVMe-TCP copy and CRC offload in
the mlx5 NIC driver as the first user.

Patches 20      Document TCP DDP offload.
Patches 21-24   Net core support for transmit offload
Patches 25-26   NVMe-TCP transmit offload support 
Patches 27-36   Mellanox NVMe-TCP transmit offload support

Testing:
=========================================
This series was tested using fio with various configurations of IO sizes, depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets. Also, we have used QEMU and gate-level simulation to verify these patches.

Future work:
=========================================
A follow-up series will introduce support for TLS in NVMe-TCP and combining the two offloads.

Ben Ben-Ishay (8):
net/mlx5e: NVMEoTCP offload initialization net/mlx5e: KLM UMR helper macros
net/mlx5e: NVMEoTCP use KLM UMRs
net/mlx5e: NVMEoTCP queue init/teardown net/mlx5e: NVMEoTCP async ddp invalidation net/mlx5e: NVMEoTCP ddp setup and resync net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload net/mlx5e: NVMEoTCP statistics

Ben Ben-ishay (2):
net/mlx5: Header file changes for nvme-tcp offload net/mlx5: Add 128B CQE for NVMEoTCP offload

Boris Pismenny (8):
net: Introduce direct data placement tcp offload
iov_iter: DDP copy to iter/pages
net: skb copy(+hash) iterators for DDP offloads net/tls: expose get_netdev_for_sock
nvme-tcp: Add DDP offload control path
nvme-tcp: Add DDP data-path
net/mlx5e: TCP flow steering for nvme-tcp
Documentation: add ULP DDP offload documentation

Or Gerlitz (1):
nvme-tcp: Deal with netdevice DOWN events

Yoray Zack (17):
nvme-tcp: RX DDGST offload
net: drop ULP DDP HW offload feature if no CSUM offload feature
net: Add ulp_ddp_pdu_info struct
net: Add to ulp_ddp support for fallback flow
net: Add MSG_DDP_CRC flag
nvme-tcp: TX DDGST offload
nvme-tcp: Mapping between Tx NVMEoTCP pdu and TCP sequence
mlx5e: make preparation in TLS code for NVMEoTCP CRC Tx offload
mlx5: Add sq state test bit for nvmeotcp
mlx5: Add support to NETIF_F_HW_TCP_DDP_CRC_TX feature net/mlx5e: NVMEoTCP DDGST TX offload TIS
net/mlx5e: NVMEoTCP DDGST Tx offload queue init/teardown net/mlx5e: NVMEoTCP DDGST TX BSF and PSV
net/mlx5e: NVMEoTCP DDGST TX Data path
net/mlx5e: NVMEoTCP DDGST TX handle OOO packets net/mlx5e: NVMEoTCP DDGST TX offload optimization net/mlx5e: NVMEoTCP DDGST TX statistics

 Documentation/networking/index.rst            |    1 +
 Documentation/networking/ulp-ddp-offload.rst  |  415 +++++
 .../net/ethernet/mellanox/mlx5/core/Kconfig   |   10 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |    2 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |   36 +-
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |    4 +-
 .../ethernet/mellanox/mlx5/core/en/params.c   |   11 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |    3 +
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   20 +-
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |    1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |    1 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |   22 +-
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      |   10 +
 .../mellanox/mlx5/core/en_accel/fs_tcp.h      |    2 +-
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     |   16 +-
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 1555 +++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    |  138 ++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        |  264 +++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.h        |   43 +
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |   80 +
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   30 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |   66 +-
 .../ethernet/mellanox/mlx5/core/en_stats.c    |   74 +
 .../ethernet/mellanox/mlx5/core/en_stats.h    |   47 +
 .../net/ethernet/mellanox/mlx5/core/en_tx.c   |   11 +
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |   17 +
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |    6 +
 drivers/nvme/host/tcp.c                       |  567 +++++-
 include/linux/mlx5/device.h                   |   44 +-
 include/linux/mlx5/mlx5_ifc.h                 |  104 +-
 include/linux/mlx5/qp.h                       |    1 +
 include/linux/netdev_features.h               |    3 +-
 include/linux/netdevice.h                     |    5 +
 include/linux/skbuff.h                        |   13 +
 include/linux/socket.h                        |    1 +
 include/linux/uio.h                           |   17 +
 include/net/inet_connection_sock.h            |    4 +
 include/net/sock.h                            |   23 +
 include/net/ulp_ddp.h                         |  192 ++
 lib/iov_iter.c                                |   55 +
 net/Kconfig                                   |   10 +
 net/core/Makefile                             |    1 +
 net/core/datagram.c                           |   48 +
 net/core/dev.c                                |    2 +
 net/core/skbuff.c                             |    8 +-
 net/core/sock.c                               |    7 +
 net/core/ulp_ddp.c                            |  235 +++
 net/ethtool/common.c                          |    1 +
 net/ipv4/tcp.c                                |    6 +
 net/ipv4/tcp_input.c                          |    8 +
 net/ipv4/tcp_ipv4.c                           |    3 +
 net/ipv4/tcp_offload.c                        |    3 +
 net/tls/tls_device.c                          |   20 +-
 53 files changed, 4192 insertions(+), 74 deletions(-)
 create mode 100644 Documentation/networking/ulp-ddp-offload.rst
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
 create mode 100644 include/net/ulp_ddp.h
 create mode 100644 net/core/ulp_ddp.c
Signature · 1 line
2.24.1
author reviewer
#1 lore raw

PATCH v5 1/36 net: Introduce direct data placement tcp offload

From: Boris Pismenny <[email protected]>

This commit introduces direct data placement offload for TCP.
This capability is accompanied by new net_device operations that configure hardware contexts. There is a context per socket, and a context per DDP opreation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments.

Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.

The SKB passed to the network stack from the driver represents data as it is on the wire, while it is pointing directly to data in destination buffers.
As a result, data from page frags should not be copied out to the linear part. To avoid needless copies, such as when using skb_condense, we mark the skb->ddp_crc bit. This bit will be used to indicate both ddp and crc offload (next patch in series).

A follow-up patch will use this interface for DDP in NVMe-TCP.

---
 include/linux/netdev_features.h    |   3 +-
 include/linux/netdevice.h          |   5 ++
 include/linux/skbuff.h             |   4 +
 include/net/inet_connection_sock.h |   4 +
 include/net/ulp_ddp.h              | 136 +++++++++++++++++++++++++++++
 net/Kconfig                        |  10 +++
 net/core/skbuff.c                  |   8 +-
 net/ethtool/common.c               |   1 +
 net/ipv4/tcp_input.c               |   8 ++
 net/ipv4/tcp_ipv4.c                |   3 +
 net/ipv4/tcp_offload.c             |   3 +
 11 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 include/net/ulp_ddp.h
include/linux/netdev_features.h +2 -1
index 2c6b9e416225..d9bd6ea26fc8 100644
@@ -14,7 +14,7 @@ typedef u64 netdev_features_t;
 enum {
 	NETIF_F_SG_BIT,			/* Scatter/gather IO. */
 	NETIF_F_IP_CSUM_BIT,		/* Can checksum TCP/UDP over IPv4. */
-	__UNUSED_NETIF_F_1,
+	NETIF_F_HW_ULP_DDP_BIT,         /* ULP direct data placement offload */
 	NETIF_F_HW_CSUM_BIT,		/* Can checksum all the packets. */
 	NETIF_F_IPV6_CSUM_BIT,		/* Can checksum TCP/UDP over IPV6 */
 	NETIF_F_HIGHDMA_BIT,		/* Can DMA to high memory. */
@@ -168,6 +168,7 @@ enum {
 #define NETIF_F_HW_HSR_TAG_RM	__NETIF_F(HW_HSR_TAG_RM)
 #define NETIF_F_HW_HSR_FWD	__NETIF_F(HW_HSR_FWD)
 #define NETIF_F_HW_HSR_DUP	__NETIF_F(HW_HSR_DUP)
+#define NETIF_F_HW_ULP_DDP	__NETIF_F(HW_ULP_DDP)
 
 /* Finds the next feature with the highest number of the range of start till 0.
  */
include/linux/netdevice.h +5 -0
index eaf5bb008aa9..cba92c2dd9c0 100644
@@ -1005,6 +1005,7 @@ struct dev_ifalias {
 
 struct devlink;
 struct tlsdev_ops;
+struct ulp_ddp_dev_ops;
 
 struct netdev_name_node {
 	struct hlist_node hlist;
@@ -2024,6 +2025,10 @@ struct net_device {
 	const struct tlsdev_ops *tlsdev_ops;
 #endif
 
+#if IS_ENABLED(CONFIG_ULP_DDP)
+	const struct ulp_ddp_dev_ops *ulp_ddp_ops;
+#endif
+
 	const struct header_ops *header_ops;
 
 	unsigned char		operstate;
include/linux/skbuff.h +4 -0
index b2db9cd9a73f..d323ecd37448 100644
@@ -689,6 +689,7 @@ typedef unsigned char *sk_buff_data_t;
  *		CHECKSUM_UNNECESSARY (max 3)
  *	@dst_pending_confirm: need to confirm neighbour
  *	@decrypted: Decrypted SKB
+ *	@ddp_crc: DDP or CRC offloaded
  *	@napi_id: id of the NAPI struct this skb came from
  *	@sender_cpu: (aka @napi_id) source CPU in XPS
  *	@secmark: security marking
@@ -870,6 +871,9 @@ struct sk_buff {
 #ifdef CONFIG_TLS_DEVICE
 	__u8			decrypted:1;
 #endif
+#ifdef CONFIG_ULP_DDP
+	__u8                    ddp_crc:1;
+#endif
 
 #ifdef CONFIG_NET_SCHED
 	__u16			tc_index;	/* traffic control index */
include/net/inet_connection_sock.h +4 -0
index b06c2d02ec84..66801ea72fb4 100644
@@ -66,6 +66,8 @@ struct inet_connection_sock_af_ops {
  * @icsk_ulp_ops	   Pluggable ULP control hook
  * @icsk_ulp_data	   ULP private data
  * @icsk_clean_acked	   Clean acked data hook
+ * @icsk_ulp_ddp_ops	   Pluggable ULP direct data placement control hook
+ * @icsk_ulp_ddp_data	   ULP direct data placement private data
  * @icsk_listen_portaddr_node	hash to the portaddr listener hashtable
  * @icsk_ca_state:	   Congestion control state
  * @icsk_retransmits:	   Number of unrecovered [RTO] timeouts
@@ -96,6 +98,8 @@ struct inet_connection_sock {
 	const struct tcp_ulp_ops  *icsk_ulp_ops;
 	void __rcu		  *icsk_ulp_data;
 	void (*icsk_clean_acked)(struct sock *sk, u32 acked_seq);
+	const struct ulp_ddp_ulp_ops  *icsk_ulp_ddp_ops;
+	void __rcu		  *icsk_ulp_ddp_data;
 	struct hlist_node         icsk_listen_portaddr_node;
 	unsigned int		  (*icsk_sync_mss)(struct sock *sk, u32 pmtu);
 	__u8			  icsk_ca_state:5,
include/net/ulp_ddp.h new file +136 -0

136 lines not on this page · raw message

net/Kconfig +10 -0
index c7392c449b25..b6f0ccbea1e3 100644
@@ -454,4 +454,14 @@ config ETHTOOL_NETLINK
 	  netlink. It provides better extensibility and some new features,
 	  e.g. notification messages.
 
+config ULP_DDP
+	bool "ULP direct data placement offload"
+	default n
+	help
+	  Direct Data Placement (DDP) offload enables ULP, such as
+	  NVMe-TCP/iSCSI, to request the NIC to place ULP payload data
+	  of a command response directly into kernel pages while
+	  calculate/verify the data digest on ULP PDU as they go through
+	  the NIC. Thus avoiding the costly per-byte overhead.
+
 endif   # if NET
net/core/skbuff.c +7 -1
index 12aabcda6db2..20add6c3f2e6 100644
@@ -71,6 +71,7 @@
 #include <net/mpls.h>
 #include <net/mptcp.h>
 #include <net/page_pool.h>
+#include <net/ulp_ddp.h>
 
 #include <linux/uaccess.h>
 #include <trace/events/skb.h>
@@ -6295,9 +6296,14 @@ EXPORT_SYMBOL(pskb_extract);
  */
 void skb_condense(struct sk_buff *skb)
 {
+	bool is_ddp = false;
+
+#ifdef CONFIG_ULP_DDP
+	is_ddp = skb->ddp_crc;
+#endif
 	if (skb->data_len) {
 		if (skb->data_len > skb->end - skb->tail ||
-		    skb_cloned(skb))
+		    skb_cloned(skb) || is_ddp)
 			return;
 
 		/* Nice, we can free page frag(s) right now */
net/ethtool/common.c +1 -0
index f9dcbad84788..d545d1525800 100644
@@ -73,6 +73,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {
 	[NETIF_F_HW_HSR_TAG_RM_BIT] =	 "hsr-tag-rm-offload",
 	[NETIF_F_HW_HSR_FWD_BIT] =	 "hsr-fwd-offload",
 	[NETIF_F_HW_HSR_DUP_BIT] =	 "hsr-dup-offload",
+	[NETIF_F_HW_ULP_DDP_BIT] =	 "ulp-ddp-offload",
 };
 
 const char
net/ipv4/tcp_input.c +8 -0
index e6ca5a1f3b59..4a7160bba09b 100644
@@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 		memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
 		nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+		nskb->ddp_crc = skb->ddp_crc;
 #endif
 		TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
 		if (list)
@@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
 				if (skb->decrypted != nskb->decrypted)
 					goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+
+				if (skb->ddp_crc != nskb->ddp_crc)
+					goto end;
 #endif
 			}
 		}
net/ipv4/tcp_ipv4.c +3 -0
index e66ad6bfe808..3d9849a39b82 100644
@@ -1830,6 +1830,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 	      TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_ECE | TCPHDR_CWR)) ||
 #ifdef CONFIG_TLS_DEVICE
 	    tail->decrypted != skb->decrypted ||
+#endif
+#ifdef CONFIG_ULP_DDP
+	    tail->ddp_crc != skb->ddp_crc ||
 #endif
 	    thtail->doff != th->doff ||
 	    memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)))
net/ipv4/tcp_offload.c +3 -0
index e09147ac9a99..96e8228d2b96 100644
@@ -262,6 +262,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
 #ifdef CONFIG_TLS_DEVICE
 	flush |= p->decrypted ^ skb->decrypted;
 #endif
+#ifdef CONFIG_ULP_DDP
+	flush |= p->ddp_crc ^ skb->ddp_crc;
+#endif
 
 	if (flush || skb_gro_receive(p, skb)) {
 		mss = 1;
Signature · 1 line
2.24.1
author reviewer
#8 lore raw

PATCH v5 2/36 iov_iter: DDP copy to iter/pages

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

When using direct data placement (DDP) the NIC writes some of the payload directly to the destination buffer, and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination we use the newly introduced routines in this commit, which check if (src == dst), and skip the copy when that's true.

As the current user for these routines is in the block layer (nvme-tcp), then we only apply the change for bio_vec. Other routines use the normal methods for copying.

---
 include/linux/uio.h | 17 ++++++++++++++
 lib/iov_iter.c      | 55 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
include/linux/uio.h +17 -0
index d3ec87706d75..a61fdb369e0e 100644
@@ -131,6 +131,9 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes,
 			 struct iov_iter *i);
 
 size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
+#ifdef CONFIG_ULP_DDP
+size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
+#endif
 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i);
 bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i);
 size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
@@ -145,6 +148,16 @@ size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
 		return _copy_to_iter(addr, bytes, i);
 }
 
+#ifdef CONFIG_ULP_DDP
+static __always_inline __must_check
+size_t ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
+{
+	if (unlikely(!check_copy_size(addr, bytes, true)))
+		return 0;
+	return _ddp_copy_to_iter(addr, bytes, i);
+}
+#endif
+
 static __always_inline __must_check
 size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)
 {
@@ -281,6 +294,10 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct io
 bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i);
 size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp,
 		struct iov_iter *i);
+#ifdef CONFIG_ULP_DDP
+size_t ddp_hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp,
+		struct iov_iter *i);
+#endif
 
 struct iovec *iovec_from_user(const struct iovec __user *uvector,
 		unsigned long nr_segs, unsigned long fast_segs,
lib/iov_iter.c +55 -0

73 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#16 lore raw

PATCH v5 3/36 net: skb copy(+hash) iterators for DDP offloads

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

This commit introduces new functions to support direct data placement (DDP) NIC offloads that avoid copying data from SKBs.

Later patches will use this for nvme-tcp DDP offload.

---
 include/linux/skbuff.h |  9 ++++++++
 net/core/datagram.c    | 48 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)
include/linux/skbuff.h +9 -0
index d323ecd37448..8c1bfd7081d1 100644
@@ -3613,6 +3613,10 @@ __poll_t datagram_poll(struct file *file, struct socket *sock,
 			   struct poll_table_struct *wait);
 int skb_copy_datagram_iter(const struct sk_buff *from, int offset,
 			   struct iov_iter *to, int size);
+#ifdef CONFIG_TCP_DDP
+int skb_ddp_copy_datagram_iter(const struct sk_buff *from, int offset,
+			       struct iov_iter *to, int size);
+#endif
 static inline int skb_copy_datagram_msg(const struct sk_buff *from, int offset,
 					struct msghdr *msg, int size)
 {
@@ -3623,6 +3627,11 @@ int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen,
 int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset,
 			   struct iov_iter *to, int len,
 			   struct ahash_request *hash);
+#ifdef CONFIG_TCP_DDP
+int skb_ddp_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset,
+					struct iov_iter *to, int len,
+					struct ahash_request *hash);
+#endif
 int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset,
 				 struct iov_iter *from, int len);
 int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm);
net/core/datagram.c +48 -0

60 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#17 lore raw

PATCH v5 4/36 net/tls: expose get_netdev_for_sock

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.

---
 include/net/sock.h   | 17 +++++++++++++++++
 net/tls/tls_device.c | 20 ++------------------
 2 files changed, 19 insertions(+), 18 deletions(-)
include/net/sock.h +17 -0
index 8bdd80027ffb..5fa2fd192d18 100644
@@ -2769,4 +2769,21 @@ void sock_set_sndtimeo(struct sock *sk, s64 secs);
 
 int sock_bind_add(struct sock *sk, struct sockaddr *addr, int addr_len);
 
+/* Assume that the socket is already connected */
+static inline struct net_device *get_netdev_for_sock(struct sock *sk, bool hold)
+{
+	struct dst_entry *dst = sk_dst_get(sk);
+	struct net_device *netdev = NULL;
+
+	if (likely(dst)) {
+		netdev = dst->dev;
+		if (hold)
+			dev_hold(netdev);
+	}
+
+	dst_release(dst);
+
+	return netdev;
+}
+
 #endif	/* _SOCK_H */
net/tls/tls_device.c +2 -18
index b932469ee69c..06aa5f1e73d1 100644
@@ -107,22 +107,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx)
 	spin_unlock_irqrestore(&tls_device_lock, flags);
 }
 
-/* We assume that the socket is already connected */
-static struct net_device *get_netdev_for_sock(struct sock *sk)
-{
-	struct dst_entry *dst = sk_dst_get(sk);
-	struct net_device *netdev = NULL;
-
-	if (likely(dst)) {
-		netdev = netdev_sk_get_lowest_dev(dst->dev, sk);
-		dev_hold(netdev);
-	}
-
-	dst_release(dst);
-
-	return netdev;
-}
-
 static void destroy_record(struct tls_record_info *record)
 {
 	int i;
@@ -1118,7 +1102,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)
 	if (skb)
 		TCP_SKB_CB(skb)->eor = 1;
 
-	netdev = get_netdev_for_sock(sk);
+	netdev = get_netdev_for_sock(sk, true);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
 		rc = -EINVAL;
@@ -1194,7 +1178,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 	if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
 		return -EOPNOTSUPP;
 
-	netdev = get_netdev_for_sock(sk);
+	netdev = get_netdev_for_sock(sk, true);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
 		return -EINVAL;
Signature · 1 line
2.24.1
author reviewer
#23 lore raw

PATCH v5 5/36 nvme-tcp: Add DDP offload control path

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the ulp_ddp_sk_add/del NDOs.

Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:

1. TCP OOO causes the NIC HW to stop the offload

2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.

3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (ulp_ddp_resync), which will update the HW,
and resume offload when all is successful.

Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via ulp_ddp_limits.

A follow-up patch introduces the data-path changes required for this offload.

---
 drivers/nvme/host/tcp.c | 180 +++++++++++++++++++++++++++++++++++++++-
 include/linux/skbuff.h  |   4 +-
 net/core/datagram.c     |   4 +-
 3 files changed, 182 insertions(+), 6 deletions(-)
drivers/nvme/host/tcp.c +178 -2
index c7bd37103cf4..f1a5520cabec 100644
@@ -14,6 +14,7 @@
 #include <linux/blk-mq.h>
 #include <crypto/hash.h>
 #include <net/busy_poll.h>
+#include <net/ulp_ddp.h>
 
 #include "nvme.h"
 #include "fabrics.h"
@@ -62,6 +63,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_ALLOCATED	= 0,
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFF_DDP	= 3,
 };
 
 enum nvme_tcp_recv_state {
@@ -111,6 +113,8 @@ struct nvme_tcp_queue {
 	void (*state_change)(struct sock *);
 	void (*data_ready)(struct sock *);
 	void (*write_space)(struct sock *);
+
+	atomic64_t  resync_req;
 };
 
 struct nvme_tcp_ctrl {
@@ -130,6 +134,8 @@ struct nvme_tcp_ctrl {
 	struct delayed_work	connect_work;
 	struct nvme_tcp_request async_req;
 	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct net_device       *offloading_netdev;
 };
 
 static LIST_HEAD(nvme_tcp_ctrl_list);
@@ -219,6 +225,167 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 	return nvme_tcp_pdu_data_left(req) <= len;
 }
 
+#ifdef CONFIG_ULP_DDP
+
+static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
+	.resync_request		= nvme_tcp_resync_request,
+};
+
+static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	struct nvme_tcp_ddp_config config = {};
+	int ret;
+
+	if (!netdev || !(netdev->features & NETIF_F_HW_ULP_DDP))
+		return -EOPNOTSUPP;
+
+	config.cfg.type		= ULP_DDP_NVME;
+	config.pfv		= NVME_TCP_PFV_1_0;
+	config.cpda		= 0;
+	config.dgst		= queue->hdr_digest ?
+		NVME_TCP_HDR_DIGEST_ENABLE : 0;
+	config.dgst		|= queue->data_digest ?
+		NVME_TCP_DATA_DIGEST_ENABLE : 0;
+	config.queue_size	= queue->queue_size;
+	config.queue_id		= nvme_tcp_queue_id(queue);
+	config.io_cpu		= queue->io_cpu;
+
+	dev_hold(netdev); /* put by unoffload_socket */
+	ret = netdev->ulp_ddp_ops->ulp_ddp_sk_add(netdev,
+						  queue->sock->sk,
+						  &config.cfg);
+	if (ret) {
+		dev_put(netdev);
+		return ret;
+	}
+
+	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
+	if (netdev->features & NETIF_F_HW_ULP_DDP)
+		set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+
+	return ret;
+}
+
+static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+
+	if (!netdev) {
+		dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n");
+		return;
+	}
+
+	clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+
+	netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk);
+
+	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL;
+	dev_put(netdev); /* held by offload_socket */
+}
+
+static int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	struct ulp_ddp_limits limits;
+	int ret = 0;
+
+	if (!netdev) {
+		dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n");
+		queue->ctrl->offloading_netdev = NULL;
+		return -ENODEV;
+	}
+
+	if ((netdev->features & NETIF_F_HW_ULP_DDP) &&
+	    netdev->ulp_ddp_ops &&
+	    netdev->ulp_ddp_ops->ulp_ddp_limits)
+		ret = netdev->ulp_ddp_ops->ulp_ddp_limits(netdev, &limits);
+	else
+		ret = -EOPNOTSUPP;
+
+	if (!ret) {
+		queue->ctrl->offloading_netdev = netdev;
+		dev_dbg_ratelimited(queue->ctrl->ctrl.device,
+				    "netdev %s offload limits: max_ddp_sgl_len %d\n",
+				    netdev->name, limits.max_ddp_sgl_len);
+		queue->ctrl->ctrl.max_segments = limits.max_ddp_sgl_len;
+		queue->ctrl->ctrl.max_hw_sectors =
+			limits.max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
+	} else {
+		queue->ctrl->offloading_netdev = NULL;
+	}
+
+	/* release the device as no offload context is established yet. */
+	dev_put(netdev);
+
+	return ret;
+}
+
+static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+				     struct sk_buff *skb, unsigned int offset)
+{
+	u64 pdu_seq = TCP_SKB_CB(skb)->seq + offset - queue->pdu_offset;
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	u64 pdu_val = (pdu_seq << 32) | ULP_DDP_RESYNC_REQ;
+	u64 resync_val;
+	u32 resync_seq;
+
+	resync_val = atomic64_read(&queue->resync_req);
+	/* Lower 32 bit flags. Check validity of the request */
+	if ((resync_val & ULP_DDP_RESYNC_REQ) == 0)
+		return;
+
+	/* Obtain and check requested sequence number: is this PDU header before the request? */
+	resync_seq = resync_val >> 32;
+	if (before(pdu_seq, resync_seq))
+		return;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return;
+	}
+
+	/**
+	 * The atomic operation gurarantees that we don't miss any NIC driver
+	 * resync requests submitted after the above checks.
+	 */
+	if (atomic64_cmpxchg(&queue->resync_req, pdu_val,
+			     pdu_val & ~ULP_DDP_RESYNC_REQ) != atomic64_read(&queue->resync_req))
+		netdev->ulp_ddp_ops->ulp_ddp_resync(netdev, queue->sock->sk, pdu_seq);
+}
+
+static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	struct nvme_tcp_queue *queue = sk->sk_user_data;
+
+	atomic64_set(&queue->resync_req,
+		     (((uint64_t)seq << 32) | flags));
+
+	return true;
+}
+
+#else
+
+static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
+{
+	return -EINVAL;
+}
+
+static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{}
+
+static int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue)
+{
+	return -EINVAL;
+}
+
+static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+				     struct sk_buff *skb, unsigned int offset)
+{}
+
+#endif
+
 static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
 		unsigned int dir)
 {
@@ -649,6 +816,9 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 	size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
 	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+		nvme_tcp_resync_response(queue, skb, *offset);
+
 	ret = skb_copy_bits(skb, *offset,
 		&pdu[queue->pdu_offset], rcv_len);
 	if (unlikely(ret))
@@ -1555,6 +1725,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
 	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
 	nvme_tcp_restore_sock_calls(queue);
 	cancel_work_sync(&queue->io_work);
+
+	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+		nvme_tcp_unoffload_socket(queue);
 }
 
 static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
@@ -1573,10 +1746,13 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
 	int ret;
 
-	if (idx)
+	if (idx) {
 		ret = nvmf_connect_io_queue(nctrl, idx, false);
-	else
+		nvme_tcp_offload_socket(&ctrl->queues[idx]);
+	} else {
 		ret = nvmf_connect_admin_queue(nctrl);
+		nvme_tcp_offload_limits(&ctrl->queues[idx]);
+	}
 
 	if (!ret) {
 		set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags);
include/linux/skbuff.h +2 -2
index 8c1bfd7081d1..55dc858ff349 100644
@@ -3613,7 +3613,7 @@ __poll_t datagram_poll(struct file *file, struct socket *sock,
 			   struct poll_table_struct *wait);
 int skb_copy_datagram_iter(const struct sk_buff *from, int offset,
 			   struct iov_iter *to, int size);
-#ifdef CONFIG_TCP_DDP
+#ifdef CONFIG_ULP_DDP
 int skb_ddp_copy_datagram_iter(const struct sk_buff *from, int offset,
 			       struct iov_iter *to, int size);
 #endif
@@ -3627,7 +3627,7 @@ int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen,
 int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset,
 			   struct iov_iter *to, int len,
 			   struct ahash_request *hash);
-#ifdef CONFIG_TCP_DDP
+#ifdef CONFIG_ULP_DDP
 int skb_ddp_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset,
 					struct iov_iter *to, int len,
 					struct ahash_request *hash);
net/core/datagram.c +2 -2
index d346fd5da22c..5ad5fb22d3f8 100644
@@ -495,7 +495,7 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset,
 	return 0;
 }
 
-#ifdef CONFIG_TCP_DDP
+#ifdef CONFIG_ULP_DDP
 /**
  *	skb_ddp_copy_and_hash_datagram_iter - Copies datagrams from skb frags to
  *	an iterator and update a hash. If the iterator and skb frag point to the
@@ -534,7 +534,7 @@ int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset,
 }
 EXPORT_SYMBOL(skb_copy_and_hash_datagram_iter);
 
-#ifdef CONFIG_TCP_DDP
+#ifdef CONFIG_ULP_DDP
 static size_t simple_ddp_copy_to_iter(const void *addr, size_t bytes,
 				      void *data __always_unused,
 				      struct iov_iter *i)
Signature · 1 line
2.24.1
author reviewer
#24 lore raw

PATCH v5 6/36 nvme-tcp: Add DDP data-path

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

Introduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id in the PDU.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.

The data-path interface contains two routines: tcp_ddp_setup/teardown. The setup provides the mapping from command_id to the request buffers, while the teardown removes this mapping.

For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the generic mq_blk completion.

---
 drivers/nvme/host/tcp.c | 150 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 138 insertions(+), 12 deletions(-)
drivers/nvme/host/tcp.c +138 -12
index f1a5520cabec..34982fb0c655 100644
@@ -57,6 +57,11 @@ struct nvme_tcp_request {
 	size_t			offset;
 	size_t			data_sent;
 	enum nvme_tcp_send_state state;
+
+	bool			offloaded;
+	struct ulp_ddp_io	ddp;
+	__le16			status;
+	union nvme_result	result;
 };
 
 enum nvme_tcp_queue_flags {
@@ -225,13 +230,76 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 	return nvme_tcp_pdu_data_left(req) <= len;
 }
 
+static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq)
+{
+	int ret;
+
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	ret = sg_alloc_table_chained(&req->ddp.sg_table, blk_rq_nr_phys_segments(rq),
+				     req->ddp.sg_table.sgl, SG_CHUNK_SIZE);
+	if (ret)
+		return -ENOMEM;
+	req->ddp.nents = blk_rq_map_sg(rq->q, rq, req->ddp.sg_table.sgl);
+	return 0;
+}
+
 #ifdef CONFIG_ULP_DDP
 
 static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+static void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
 static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
 	.resync_request		= nvme_tcp_resync_request,
+	.ddp_teardown_done	= nvme_tcp_ddp_teardown_done,
 };
 
+static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+				 u16 command_id,
+				 struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	ret = netdev->ulp_ddp_ops->ulp_ddp_teardown(netdev, queue->sock->sk,
+						    &req->ddp, rq);
+	sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+	return ret;
+}
+
+static void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{
+	struct request *rq = ddp_ctx;
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+	if (!nvme_try_complete_req(rq, req->status, req->result))
+		nvme_complete_rq(rq);
+}
+
+static int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+			      u16 command_id,
+			      struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	if (!test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) ||
+	    !blk_rq_nr_phys_segments(rq) || !(rq_data_dir(rq) == READ))
+		return -EINVAL;
+
+	req->ddp.command_id = command_id;
+	ret = nvme_tcp_req_map_sg(req, rq);
+	if (ret)
+		return -ENOMEM;
+
+	ret = netdev->ulp_ddp_ops->ulp_ddp_setup(netdev,
+						 queue->sock->sk,
+						 &req->ddp);
+	if (!ret)
+		req->offloaded = true;
+	return ret;
+}
+
 static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
 {
 	struct net_device *netdev = queue->ctrl->offloading_netdev;
@@ -342,7 +410,7 @@ static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
 		return;
 
 	if (unlikely(!netdev)) {
-		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n");
 		return;
 	}
 
@@ -367,6 +435,20 @@ static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
 
 #else
 
+static int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+			      u16 command_id,
+			      struct request *rq)
+{
+	return -EINVAL;
+}
+
+static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+				 u16 command_id,
+				 struct request *rq)
+{
+	return -EINVAL;
+}
+
 static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
 {
 	return -EINVAL;
@@ -650,6 +732,24 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
 	queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work);
 }
 
+static void nvme_tcp_complete_request(struct request *rq,
+				      __le16 status,
+				      union nvme_result result,
+				      __u16 command_id)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_queue *queue = req->queue;
+
+	if (req->offloaded) {
+		req->status = status;
+		req->result = result;
+		nvme_tcp_teardown_ddp(queue, command_id, rq);
+	} else {
+		if (!nvme_try_complete_req(rq, status, result))
+			nvme_complete_rq(rq);
+	}
+}
+
 static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 		struct nvme_completion *cqe)
 {
@@ -664,10 +764,8 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 		return -EINVAL;
 	}
 
-	if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
-		nvme_complete_rq(rq);
+	nvme_tcp_complete_request(rq, cqe->status, cqe->result, cqe->command_id);
 	queue->nr_cqe++;
-
 	return 0;
 }
 
@@ -863,10 +961,39 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
 static inline void nvme_tcp_end_request(struct request *rq, u16 status)
 {
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_queue *queue = req->queue;
+	struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
 	union nvme_result res = {};
 
-	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
-		nvme_complete_rq(rq);
+	nvme_tcp_complete_request(rq, cpu_to_le16(status << 1), res, pdu->command_id);
+}
+
+
+static int nvme_tcp_consume_skb(struct nvme_tcp_queue *queue, struct sk_buff *skb,
+				unsigned int *offset, struct iov_iter *iter, int recv_len)
+{
+	int ret;
+
+#ifdef CONFIG_ULP_DDP
+	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) {
+		if (queue->data_digest)
+			ret = skb_ddp_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len,
+					queue->rcv_hash);
+		else
+			ret = skb_ddp_copy_datagram_iter(skb, *offset, iter, recv_len);
+	} else {
+#endif
+		if (queue->data_digest)
+			ret = skb_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len,
+					queue->rcv_hash);
+		else
+			ret = skb_copy_datagram_iter(skb, *offset, iter, recv_len);
+#ifdef CONFIG_ULP_DDP
+	}
+#endif
+
+	return ret;
 }
 
 static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -913,12 +1040,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 		recv_len = min_t(size_t, recv_len,
 				iov_iter_count(&req->iter));
 
-		if (queue->data_digest)
-			ret = skb_copy_and_hash_datagram_iter(skb, *offset,
-				&req->iter, recv_len, queue->rcv_hash);
-		else
-			ret = skb_copy_datagram_iter(skb, *offset,
-					&req->iter, recv_len);
+		ret = nvme_tcp_consume_skb(queue, skb, offset, &req->iter, recv_len);
 		if (ret) {
 			dev_err(queue->ctrl->ctrl.device,
 				"queue %d failed to copy request %#x data",
@@ -1142,6 +1264,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	bool inline_data = nvme_tcp_has_inline_data(req);
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) + hdgst - req->offset;
+	struct request *rq = blk_mq_rq_from_pdu(req);
 	int flags = MSG_DONTWAIT;
 	int ret;
 
@@ -1150,6 +1273,8 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	else
 		flags |= MSG_EOR;
 
+	nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq);
+
 	if (queue->hdr_digest && !req->offset)
 		nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu));
 
@@ -2486,6 +2611,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,
 	if (req->curr_bio && req->data_len)
 		nvme_tcp_init_iter(req, rq_data_dir(rq));
 
+	req->offloaded = false;
 	if (rq_data_dir(rq) == WRITE &&
 	    req->data_len <= nvme_tcp_inline_data_size(queue))
 		req->pdu_len = req->data_len;
Signature · 1 line
2.24.1
author reviewer
#26 lore raw

PATCH v5 8/36 nvme-tcp: Deal with netdevice DOWN events

In reply to Boris Pismenny

From: Or Gerlitz <[email protected]>

For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.

These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroys them.

Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.

We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.

This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.

---
 drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
drivers/nvme/host/tcp.c +39 -0
index b23fdbb4fd8b..b338cd2d9f65 100644
@@ -147,6 +147,7 @@ struct nvme_tcp_ctrl {
 
 static LIST_HEAD(nvme_tcp_ctrl_list);
 static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
+static struct notifier_block nvme_tcp_netdevice_nb;
 static struct workqueue_struct *nvme_tcp_wq;
 static const struct blk_mq_ops nvme_tcp_mq_ops;
 static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -2948,6 +2949,30 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
 	return ERR_PTR(ret);
 }
 
+static int nvme_tcp_netdev_event(struct notifier_block *this,
+				 unsigned long event, void *ptr)
+{
+	struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
+	struct nvme_tcp_ctrl *ctrl;
+
+	switch (event) {
+	case NETDEV_GOING_DOWN:
+		mutex_lock(&nvme_tcp_ctrl_mutex);
+		list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) {
+			if (ndev != ctrl->offloading_netdev)
+				continue;
+			nvme_tcp_error_recovery(&ctrl->ctrl);
+		}
+		mutex_unlock(&nvme_tcp_ctrl_mutex);
+		flush_workqueue(nvme_reset_wq);
+		/*
+		 * The associated controllers teardown has completed, ddp contexts
+		 * were also torn down so we should be safe to continue...
+		 */
+	}
+	return NOTIFY_DONE;
+}
+
 static struct nvmf_transport_ops nvme_tcp_transport = {
 	.name		= "tcp",
 	.module		= THIS_MODULE,
@@ -2962,13 +2987,26 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
 
 static int __init nvme_tcp_init_module(void)
 {
+	int ret;
+
 	nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq",
 			WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
 	if (!nvme_tcp_wq)
 		return -ENOMEM;
 
+	nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event;
+	ret = register_netdevice_notifier(&nvme_tcp_netdevice_nb);
+	if (ret) {
+		pr_err("failed to register netdev notifier\n");
+		goto out_err_reg_notifier;
+	}
+
 	nvmf_register_transport(&nvme_tcp_transport);
 	return 0;
+
+out_err_reg_notifier:
+	destroy_workqueue(nvme_tcp_wq);
+	return ret;
 }
 
 static void __exit nvme_tcp_cleanup_module(void)
@@ -2976,6 +3014,7 @@ static void __exit nvme_tcp_cleanup_module(void)
 	struct nvme_tcp_ctrl *ctrl;
 
 	nvmf_unregister_transport(&nvme_tcp_transport);
+	unregister_netdevice_notifier(&nvme_tcp_netdevice_nb);
 
 	mutex_lock(&nvme_tcp_ctrl_mutex);
 	list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list)
Signature · 1 line
2.24.1
author reviewer
#25 lore raw

PATCH v5 7/36 nvme-tcp: RX DDGST offload

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

Enable rx side of DDGST offload when supported.

At the end of the capsule, check if all the skb bits are on, and if not recalculate the DDGST in SW and check it.

We reworked the receive-side DDGST calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload DDGST calculation.

---
 drivers/nvme/host/tcp.c | 86 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 15 deletions(-)
drivers/nvme/host/tcp.c +71 -15
index 34982fb0c655..b23fdbb4fd8b 100644
@@ -69,6 +69,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
 	NVME_TCP_Q_OFF_DDP	= 3,
+	NVME_TCP_Q_OFF_DDGST_RX = 4,
 };
 
 enum nvme_tcp_recv_state {
@@ -96,6 +97,7 @@ struct nvme_tcp_queue {
 	size_t			data_remaining;
 	size_t			ddgst_remaining;
 	unsigned int		nr_cqe;
+	bool			ddgst_valid;
 
 	/* send state */
 	struct nvme_tcp_request *request;
@@ -230,6 +232,22 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 	return nvme_tcp_pdu_data_left(req) <= len;
 }
 
+static inline bool nvme_tcp_ddp_ddgst_ok(struct nvme_tcp_queue *queue)
+{
+	return queue->ddgst_valid;
+}
+
+static inline void nvme_tcp_ddp_ddgst_update(struct nvme_tcp_queue *queue,
+					     struct sk_buff *skb)
+{
+	if (queue->ddgst_valid)
+#ifdef CONFIG_ULP_DDP
+		queue->ddgst_valid = skb->ddp_crc;
+#else
+		queue->ddgst_valid = false;
+#endif
+}
+
 static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq)
 {
 	int ret;
@@ -243,6 +261,26 @@ static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq)
 	return 0;
 }
 
+static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash,
+				      struct request *rq)
+{
+	struct nvme_tcp_request *req;
+
+	if (!rq)
+		return;
+
+	req = blk_mq_rq_to_pdu(rq);
+
+	if (!req->offloaded && nvme_tcp_req_map_sg(req, rq))
+		return;
+
+	crypto_ahash_init(hash);
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	ahash_request_set_crypt(hash, req->ddp.sg_table.sgl, NULL,
+				le32_to_cpu(req->data_len));
+	crypto_ahash_update(hash);
+}
+
 #ifdef CONFIG_ULP_DDP
 
 static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
@@ -330,8 +368,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
 	}
 
 	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
-	if (netdev->features & NETIF_F_HW_ULP_DDP)
+	if (netdev->features & NETIF_F_HW_ULP_DDP) {
 		set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+		set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
+	}
 
 	return ret;
 }
@@ -346,6 +386,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
 	}
 
 	clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+	clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
 
 	netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk);
 
@@ -721,6 +762,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
 	queue->pdu_offset = 0;
 	queue->data_remaining = -1;
 	queue->ddgst_remaining = 0;
+	queue->ddgst_valid = true;
 }
 
 static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
@@ -914,7 +956,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 	size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
 	int ret;
 
-	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) ||
+	    test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
 		nvme_tcp_resync_response(queue, skb, *offset);
 
 	ret = skb_copy_bits(skb, *offset,
@@ -977,14 +1020,14 @@ static int nvme_tcp_consume_skb(struct nvme_tcp_queue *queue, struct sk_buff *sk
 
 #ifdef CONFIG_ULP_DDP
 	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) {
-		if (queue->data_digest)
+		if (queue->data_digest && !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
 			ret = skb_ddp_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len,
 					queue->rcv_hash);
 		else
 			ret = skb_ddp_copy_datagram_iter(skb, *offset, iter, recv_len);
 	} else {
 #endif
-		if (queue->data_digest)
+		if (queue->data_digest && !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
 			ret = skb_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len,
 					queue->rcv_hash);
 		else
@@ -1003,6 +1046,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 	struct nvme_tcp_request *req;
 	struct request *rq;
 
+	if (queue->data_digest && test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
+		nvme_tcp_ddp_ddgst_update(queue, skb);
 	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
@@ -1055,7 +1100,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
 	if (!queue->data_remaining) {
 		if (queue->data_digest) {
-			nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
 			queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
 		} else {
 			if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
@@ -1076,8 +1120,12 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
 	char *ddgst = (char *)&queue->recv_ddgst;
 	size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining);
 	off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
+	bool offload_fail, offload_en;
+	struct request *rq = NULL;
 	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
+		nvme_tcp_ddp_ddgst_update(queue, skb);
 	ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len);
 	if (unlikely(ret))
 		return ret;
@@ -1088,18 +1136,25 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
 	if (queue->ddgst_remaining)
 		return 0;
 
-	if (queue->recv_ddgst != queue->exp_ddgst) {
-		dev_err(queue->ctrl->ctrl.device,
-			"data digest error: recv %#x expected %#x\n",
-			le32_to_cpu(queue->recv_ddgst),
-			le32_to_cpu(queue->exp_ddgst));
-		return -EIO;
+	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+
+	offload_fail = !nvme_tcp_ddp_ddgst_ok(queue);
+	offload_en = test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
+	if (!offload_en || offload_fail) {
+		if (offload_en && offload_fail)  // software-fallback
+			nvme_tcp_ddp_ddgst_recalc(queue->rcv_hash, rq);
+
+		nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
+		if (queue->recv_ddgst != queue->exp_ddgst) {
+			dev_err(queue->ctrl->ctrl.device,
+				"data digest error: recv %#x expected %#x\n",
+				le32_to_cpu(queue->recv_ddgst),
+				le32_to_cpu(queue->exp_ddgst));
+			return -EIO;
+		}
 	}
 
 	if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
-		struct request *rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue),
-						pdu->command_id);
-
 		nvme_tcp_end_request(rq, NVME_SC_SUCCESS);
 		queue->nr_cqe++;
 	}
@@ -1851,7 +1906,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
 	nvme_tcp_restore_sock_calls(queue);
 	cancel_work_sync(&queue->io_work);
 
-	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) ||
+	    test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
 		nvme_tcp_unoffload_socket(queue);
 }
 
Signature · 1 line
2.24.1
author reviewer
#28 lore raw

PATCH v5 10/36 net/mlx5: Add 128B CQE for NVMEoTCP offload

In reply to Boris Pismenny

From: Ben Ben-ishay <[email protected]>

Add the NVMEoTCP offload definition and access functions for 128B CQEs.

---
 include/linux/mlx5/device.h | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)
include/linux/mlx5/device.h +35 -1
index a42e47f91327..412d0982ee46 100644
@@ -794,7 +794,7 @@ struct mlx5_err_cqe {
 
 struct mlx5_cqe64 {
 	u8		tls_outer_l3_tunneled;
-	u8		rsvd0;
+	u8		nvmetcp;
 	__be16		wqe_id;
 	u8		lro_tcppsh_abort_dupack;
 	u8		lro_min_ttl;
@@ -827,6 +827,19 @@ struct mlx5_cqe64 {
 	u8		op_own;
 };
 
+struct mlx5e_cqe128 {
+	__be16		cclen;
+	__be16		hlen;
+	union {
+		__be32		resync_tcp_sn;
+		__be32		ccoff;
+	};
+	__be16		ccid;
+	__be16		rsvd8;
+	u8		rsvd12[52];
+	struct mlx5_cqe64 cqe64;
+};
+
 struct mlx5_mini_cqe8 {
 	union {
 		__be32 rx_hash_result;
@@ -857,6 +870,27 @@ enum {
 
 #define MLX5_MINI_CQE_ARRAY_SIZE 8
 
+static inline bool cqe_is_nvmeotcp_resync(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 6) & 0x1);
+}
+
+static inline bool cqe_is_nvmeotcp_crcvalid(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 5) & 0x1);
+}
+
+static inline bool cqe_is_nvmeotcp_zc(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 4) & 0x1);
+}
+
+/* check if cqe is zc or crc or resync */
+static inline bool cqe_is_nvmeotcp(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 4) & 0x7);
+}
+
 static inline u8 mlx5_get_cqe_format(struct mlx5_cqe64 *cqe)
 {
 	return (cqe->op_own >> 2) & 0x3;
Signature · 1 line
2.24.1
author reviewer
#29 lore raw

PATCH v5 11/36 net/mlx5e: TCP flow steering for nvme-tcp

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

Both nvme-tcp and tls require tcp flow steering. Compile it for both of them. Additionally, use reference counting to allocate/free TCP flow steering.

---
 drivers/net/ethernet/mellanox/mlx5/core/en/fs.h        |  4 ++--
 .../net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c  | 10 ++++++++++
 .../net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h  |  2 +-
 3 files changed, 13 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +2 -2
index 1d5ce07b83f4..8690919f2cde 100644
@@ -132,7 +132,7 @@ enum {
 	MLX5E_INNER_TTC_FT_LEVEL,
 	MLX5E_FS_TT_UDP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
 	MLX5E_FS_TT_ANY_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
 	MLX5E_ACCEL_FS_TCP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
 #endif
 #ifdef CONFIG_MLX5_EN_ARFS
@@ -231,7 +231,7 @@ struct mlx5e_flow_steering {
 #ifdef CONFIG_MLX5_EN_ARFS
 	struct mlx5e_arfs_tables       *arfs;
 #endif
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
 	struct mlx5e_accel_fs_tcp      *accel_tcp;
 #endif
 	struct mlx5e_fs_udp            *udp;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +10 -0
index e51f60b55daa..21341a92f355 100644
@@ -14,6 +14,7 @@ enum accel_fs_tcp_type {
 struct mlx5e_accel_fs_tcp {
 	struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES];
 	struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES];
+	refcount_t		ref_count;
 };
 
 static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
@@ -337,6 +338,7 @@ static int accel_fs_tcp_enable(struct mlx5e_priv *priv)
 			return err;
 		}
 	}
+	refcount_set(&priv->fs.accel_tcp->ref_count, 1);
 	return 0;
 }
 
@@ -360,6 +362,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)
 	if (!priv->fs.accel_tcp)
 		return;
 
+	if (!refcount_dec_and_test(&priv->fs.accel_tcp->ref_count))
+		return;
+
 	accel_fs_tcp_disable(priv);
 
 	for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++)
@@ -376,6 +381,11 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)
 	if (!MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, ft_field_support.outer_ip_version))
 		return -EOPNOTSUPP;
 
+	if (priv->fs.accel_tcp) {
+		refcount_inc(&priv->fs.accel_tcp->ref_count);
+		return 0;
+	}
+
 	priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
 	if (!priv->fs.accel_tcp)
 		return -ENOMEM;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h +1 -1
index 589235824543..8aff9298183c 100644
@@ -6,7 +6,7 @@
 
 #include "en.h"
 
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
 int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv);
 void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv);
 struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
Signature · 1 line
2.24.1
author reviewer
#27 lore raw

PATCH v5 9/36 net/mlx5: Header file changes for nvme-tcp offload

In reply to Boris Pismenny

From: Ben Ben-ishay <[email protected]>

Add the necessary infrastructure for NVMEoTCP offload:

- Add nvmeocp_en + nvmeotcp_crc_en bit to the TIR for identify NVMEoTCP offload flow And tag_buffer_id that will be used by the connected nvmeotcp_queues
- Add new CQE field that will be used to pass scattered data information to SW
- Add new capability to HCA_CAP that represnts the NVMEoTCP offload ability

---
 include/linux/mlx5/device.h   |   8 +++
 include/linux/mlx5/mlx5_ifc.h | 101 +++++++++++++++++++++++++++++++++-
 include/linux/mlx5/qp.h       |   1 +
 3 files changed, 107 insertions(+), 3 deletions(-)
include/linux/mlx5/device.h +8 -0
index 0025913505ab..a42e47f91327 100644
@@ -263,6 +263,7 @@ enum {
 enum {
 	MLX5_MKEY_MASK_LEN		= 1ull << 0,
 	MLX5_MKEY_MASK_PAGE_SIZE	= 1ull << 1,
+	MLX5_MKEY_MASK_XLT_OCT_SIZE     = 1ull << 2,
 	MLX5_MKEY_MASK_START_ADDR	= 1ull << 6,
 	MLX5_MKEY_MASK_PD		= 1ull << 7,
 	MLX5_MKEY_MASK_EN_RINVAL	= 1ull << 8,
@@ -1179,6 +1180,7 @@ enum mlx5_cap_type {
 	MLX5_CAP_VDPA_EMULATION = 0x13,
 	MLX5_CAP_DEV_EVENT = 0x14,
 	MLX5_CAP_IPSEC,
+	MLX5_CAP_DEV_NVMEOTCP = 0x19,
 	MLX5_CAP_GENERAL_2 = 0x20,
 	/* NUM OF CAP Types */
 	MLX5_CAP_NUM
@@ -1409,6 +1411,12 @@ enum mlx5_qcam_feature_groups {
 #define MLX5_CAP_IPSEC(mdev, cap)\
 	MLX5_GET(ipsec_cap, (mdev)->caps.hca_cur[MLX5_CAP_IPSEC], cap)
 
+#define MLX5_CAP_DEV_NVMEOTCP(mdev, cap)\
+	MLX5_GET(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap)
+
+#define MLX5_CAP64_NVMEOTCP(mdev, cap)\
+	MLX5_GET64(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap)
+
 enum {
 	MLX5_CMD_STAT_OK			= 0x0,
 	MLX5_CMD_STAT_INT_ERR			= 0x1,
include/linux/mlx5/mlx5_ifc.h +98 -3

159 lines not on this page · raw message

include/linux/mlx5/qp.h +1 -0
index b7deb790f257..dfd744a564d5 100644
@@ -220,6 +220,7 @@ struct mlx5_wqe_ctrl_seg {
 #define MLX5_WQE_CTRL_OPCODE_MASK 0xff
 #define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00
 #define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8
+#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8
 
 enum {
 	MLX5_ETH_WQE_L3_INNER_CSUM      = 1 << 4,
Signature · 1 line
2.24.1
author reviewer
#30 lore raw

PATCH v5 12/36 net/mlx5e: NVMEoTCP offload initialization

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

This commit introduce the initialization blocks for NVMEoTCP offload:
- Use 128B CQEs when NVME-TCP offload is enabled.
- Use a dedicated icosq for NVME-TCP work. This list of SQ is unique in the sense that it is driven directly by the NVME-TCP layer to submit and invalidate ddp requests.
- Query nvmeotcp capabilities

---
 .../net/ethernet/mellanox/mlx5/core/Kconfig   |  10 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  11 +
 .../ethernet/mellanox/mlx5/core/en/params.c   |  11 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |   3 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |   9 +-
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 196 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    | 117 +++++++++++
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  26 +++
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  17 ++
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   6 +
 11 files changed, 405 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +10 -0
index e1a5a79e27c7..e6079ff2e917 100644
@@ -233,3 +233,13 @@ config MLX5_SF_MANAGER
 	port is managed through devlink.  A subfunction supports RDMA, netdevice
 	and vdpa device. It is similar to a SRIOV VF but it doesn't require
 	SRIOV support.
+
+config MLX5_EN_NVMEOTCP
+	bool "NVMEoTCP accelaration"
+	depends on MLX5_CORE_EN
+	depends on ULP_DDP=y
+	default n
+	help
+	Build support for NVMEoTCP accelaration in the NIC.
+	Note: Support for hardware with this capability needs to be selected
+	for this option to become available.
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -0
index b5072a3a2585..0ae9e5e38ec7 100644
@@ -101,3 +101,5 @@ mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o
 # SF manager
 #
 mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/hw_table.o sf/devlink.o
+
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +11 -0
index b1b51bbba054..1233ebcf311b 100644
@@ -270,6 +270,10 @@ struct mlx5e_params {
 	unsigned int sw_mtu;
 	int hard_mtu;
 	bool ptp_rx;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	bool nvmeotcp;
+	bool crc_rx_offload;
+#endif
 };
 
 enum {
@@ -678,6 +682,10 @@ struct mlx5e_channel {
 	struct mlx5e_txqsq         sq[MLX5E_MAX_NUM_TC];
 	struct mlx5e_icosq         icosq;   /* internal control operations */
 	struct mlx5e_txqsq __rcu * __rcu *qos_sqs;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct list_head	   list_nvmeotcpsq;   /* nvmeotcp umrs  */
+	spinlock_t                 nvmeotcp_icosq_lock;
+#endif
 	bool                       xdp;
 	struct napi_struct         napi;
 	struct device             *pdev;
@@ -886,6 +894,9 @@ struct mlx5e_priv {
 #endif
 #ifdef CONFIG_MLX5_EN_TLS
 	struct mlx5e_tls          *tls;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct mlx5e_nvmeotcp      *nvmeotcp;
 #endif
 	struct devlink_health_reporter *tx_reporter;
 	struct devlink_health_reporter *rx_reporter;
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +9 -2
index 150c8e82c738..a84508425e47 100644
@@ -437,7 +437,8 @@ static void mlx5e_build_common_cq_param(struct mlx5_core_dev *mdev,
 	void *cqc = param->cqc;
 
 	MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
-	if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128)
+	if (MLX5_CAP_GEN(mdev, cqe_128_always) &&
+	   (cache_line_size() >= 128 || param->force_cqe128))
 		MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD);
 }
 
@@ -450,6 +451,12 @@ static void mlx5e_build_rx_cq_param(struct mlx5_core_dev *mdev,
 	void *cqc = param->cqc;
 	u8 log_cq_size;
 
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	/* nvme-tcp offload mandates 128 byte cqes */
+	param->force_cqe128 |= (params->nvmeotcp|| params->crc_rx_offload);
+#endif
+
+
 	switch (params->rq_wq_type) {
 	case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
 		log_cq_size = mlx5e_mpwqe_get_log_rq_size(params, xsk) +
@@ -620,7 +627,7 @@ static u8 mlx5e_build_async_icosq_log_wq_sz(struct mlx5_core_dev *mdev)
 	return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
 }
 
-static void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
+void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
 				    u8 log_wq_size,
 				    struct mlx5e_sq_param *param)
 {
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +3 -0
index e9593f5f0661..4f232ba726ec 100644
@@ -16,6 +16,7 @@ struct mlx5e_cq_param {
 	struct mlx5_wq_param       wq;
 	u16                        eq_ix;
 	u8                         cq_period_mode;
+	bool                       force_cqe128;
 };
 
 struct mlx5e_rq_param {
@@ -147,6 +148,8 @@ int mlx5e_build_channel_param(struct mlx5_core_dev *mdev,
 			      struct mlx5e_params *params,
 			      u16 q_counter,
 			      struct mlx5e_channel_param *cparam);
+void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
+			     u8 log_wq_size,struct mlx5e_sq_param *param);
 
 u16 mlx5e_calc_sq_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params);
 int mlx5e_validate_params(struct mlx5_core_dev *mdev, struct mlx5e_params *params);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +8 -1
index d964665eaa63..b9404366e6e8 100644
@@ -39,6 +39,7 @@
 #include "en_accel/ipsec_rxtx.h"
 #include "en_accel/tls.h"
 #include "en_accel/tls_rxtx.h"
+#include "en_accel/nvmeotcp.h"
 #include "en.h"
 #include "en/txrx.h"
 
@@ -195,11 +196,17 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,
 
 static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv)
 {
-	return mlx5e_ktls_init_rx(priv);
+	int tls, nvmeotcp;
+
+	tls = mlx5e_ktls_init_rx(priv);
+	nvmeotcp = mlx5e_nvmeotcp_init_rx(priv);
+
+	return tls && nvmeotcp;
 }
 
 static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv)
 {
+	mlx5e_nvmeotcp_cleanup_rx(priv);
 	mlx5e_ktls_cleanup_rx(priv);
 }
 #endif /* __MLX5E_EN_ACCEL_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c new file +196 -0

196 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h new file +117 -0

117 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_main.c +26 -0
index 414a73d16619..ecb12c7fdb7d 100644
@@ -47,6 +47,7 @@
 #include "en_accel/ipsec.h"
 #include "en_accel/en_accel.h"
 #include "en_accel/tls.h"
+#include "en_accel/nvmeotcp.h"
 #include "accel/ipsec.h"
 #include "accel/tls.h"
 #include "lib/vxlan.h"
@@ -2007,6 +2008,10 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 	c->aff_mask = irq_get_effective_affinity_mask(irq);
 	c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
 
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	INIT_LIST_HEAD(&c->list_nvmeotcpsq);
+	spin_lock_init(&c->nvmeotcp_icosq_lock);
+#endif
 	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll, 64);
 
 	err = mlx5e_open_queues(c, params, cparam);
@@ -3820,6 +3825,9 @@ int mlx5e_set_features(struct net_device *netdev, netdev_features_t features)
 	err |= MLX5E_HANDLE_FEATURE(NETIF_F_NTUPLE, set_feature_arfs);
 #endif
 	err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_TLS_RX, mlx5e_ktls_set_feature_rx);
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_ULP_DDP, set_feature_nvme_tcp);
+#endif
 
 	if (err) {
 		netdev->features = oper_features;
@@ -3858,6 +3866,17 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev,
 		features &= ~NETIF_F_RXHASH;
 		if (netdev->features & NETIF_F_RXHASH)
 			netdev_warn(netdev, "Disabling rxhash, not supported when CQE compress is active\n");
+
+		features &= ~NETIF_F_HW_ULP_DDP;
+		if (netdev->features & NETIF_F_HW_ULP_DDP)
+			netdev_warn(netdev, "Disabling tcp-ddp offload, not supported when CQE compress is active\n");
+
+	}
+
+	if (netdev->features & NETIF_F_LRO) {
+		features &= ~NETIF_F_HW_ULP_DDP;
+		if (netdev->features & NETIF_F_HW_ULP_DDP)
+			netdev_warn(netdev, "Disabling tcp-ddp offload, not supported when LRO is active\n");
 	}
 
 	if (mlx5e_is_uplink_rep(priv)) {
@@ -4890,6 +4909,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)
 	mlx5e_set_netdev_dev_addr(netdev);
 	mlx5e_ipsec_build_netdev(priv);
 	mlx5e_tls_build_netdev(priv);
+	mlx5e_nvmeotcp_build_netdev(priv);
 }
 
 void mlx5e_create_q_counters(struct mlx5e_priv *priv)
@@ -4950,6 +4970,10 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,
 	if (err)
 		mlx5_core_err(mdev, "TLS initialization failed, %d\n", err);
 
+	err = mlx5e_nvmeotcp_init(priv);
+	if (err)
+		mlx5_core_err(mdev, "NVMEoTCP initialization failed, %d\n", err);
+
 	dl_port = mlx5e_devlink_get_dl_port(priv);
 	if (dl_port->registered)
 		mlx5e_health_create_reporters(priv);
@@ -4963,6 +4987,8 @@ static void mlx5e_nic_cleanup(struct mlx5e_priv *priv)
 
 	if (dl_port->registered)
 		mlx5e_health_destroy_reporters(priv);
+
+	mlx5e_nvmeotcp_cleanup(priv);
 	mlx5e_tls_cleanup(priv);
 	mlx5e_ipsec_cleanup(priv);
 }
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +17 -0
index 833be29170a1..3fc11b71de67 100644
@@ -37,6 +37,7 @@
 #include "en/xsk/rx.h"
 #include "en/xsk/tx.h"
 #include "en_accel/ktls_txrx.h"
+#include "en_accel/nvmeotcp.h"
 
 static inline bool mlx5e_channel_no_affinity_change(struct mlx5e_channel *c)
 {
@@ -119,6 +120,10 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	struct mlx5e_txqsq __rcu **qos_sqs;
 	struct mlx5e_rq *xskrq = &c->xskrq;
 	struct mlx5e_rq *rq = &c->rq;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct mlx5e_nvmeotcp_sq *nvmeotcp_sq;
+	struct list_head *cur;
+#endif
 	bool aff_change = false;
 	bool busy_xsk = false;
 	bool busy = false;
@@ -171,6 +176,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 		 * queueing more WQEs and overflowing the async ICOSQ.
 		 */
 		clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state);
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	list_for_each(cur, &c->list_nvmeotcpsq) {
+		nvmeotcp_sq = list_entry(cur, struct mlx5e_nvmeotcp_sq, list);
+		mlx5e_poll_ico_cq(&nvmeotcp_sq->icosq.cq);
+	}
+#endif
 
 	/* Keep after async ICOSQ CQ poll */
 	if (unlikely(mlx5e_ktls_rx_pending_resync_list(c, budget)))
@@ -223,6 +234,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	mlx5e_cq_arm(&rq->cq);
 	mlx5e_cq_arm(&c->icosq.cq);
 	mlx5e_cq_arm(&c->async_icosq.cq);
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	list_for_each(cur, &c->list_nvmeotcpsq) {
+		nvmeotcp_sq = list_entry(cur, struct mlx5e_nvmeotcp_sq, list);
+		mlx5e_cq_arm(&nvmeotcp_sq->icosq.cq);
+	}
+#endif
 	mlx5e_cq_arm(&c->xdpsq.cq);
 
 	if (xsk_open) {
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
index 016d26f809a5..a8a14c15a61f 100644
@@ -262,6 +262,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 			return err;
 	}
 
+	if (MLX5_CAP_GEN(dev, nvmeotcp)) {
+		err = mlx5_core_get_caps(dev, MLX5_CAP_DEV_NVMEOTCP);
+		if (err)
+			return err;
+	}
+
 	return 0;
 }
 
Signature · 1 line
2.24.1
author reviewer
#31 lore raw

PATCH v5 13/36 net/mlx5e: KLM UMR helper macros

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

Add helper macros for posting KLM UMR WQE.

---
 drivers/net/ethernet/mellanox/mlx5/core/en.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en.h +18 -0
index 1233ebcf311b..5bc38002d136 100644
@@ -155,6 +155,24 @@ struct page_pool;
 #define MLX5E_UMR_WQEBBS \
 	(DIV_ROUND_UP(MLX5E_UMR_WQE_INLINE_SZ, MLX5_SEND_WQE_BB))
 
+#define KLM_ALIGNMENT 4
+#define MLX5E_KLM_UMR_WQE_SZ(sgl_len)\
+	(sizeof(struct mlx5e_umr_wqe) +\
+	(sizeof(struct mlx5_klm) * (sgl_len)))
+
+#define MLX5E_KLM_UMR_WQEBBS(sgl_len)\
+	(DIV_ROUND_UP(MLX5E_KLM_UMR_WQE_SZ(sgl_len), MLX5_SEND_WQE_BB))
+
+#define MLX5E_KLM_UMR_DS_CNT(sgl_len)\
+	DIV_ROUND_UP(MLX5E_KLM_UMR_WQE_SZ(sgl_len), MLX5_SEND_WQE_DS)
+
+#define MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size)\
+	(((wqe_size) - sizeof(struct mlx5e_umr_wqe)) / sizeof(struct mlx5_klm))
+
+#define MLX5E_KLM_ENTRIES_PER_WQE(wqe_size)\
+	(MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size) -\
+			(MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size) % KLM_ALIGNMENT))
+
 #define MLX5E_MSG_LEVEL			NETIF_MSG_LINK
 
 #define mlx5e_dbg(mlevel, priv, format, ...)                    \
Signature · 1 line
2.24.1
author reviewer
#32 lore raw

PATCH v5 14/36 net/mlx5e: NVMEoTCP use KLM UMRs

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

NVMEoTCP offload uses buffer registration for ddp operation, every request comprises from SG list that might have elements with size > 4K, thus the appropriate way to perform buffer registration is with KLM UMRs.

---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |   5 +-
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   3 +
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 116 ++++++++++++++++++
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |  12 ++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  12 +-
 5 files changed, 145 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
drivers/net/ethernet/mellanox/mlx5/core/en.h +4 -1
index 5bc38002d136..d88ecd04d5da 100644
@@ -238,7 +238,10 @@ struct mlx5e_umr_wqe {
 	struct mlx5_wqe_ctrl_seg       ctrl;
 	struct mlx5_wqe_umr_ctrl_seg   uctrl;
 	struct mlx5_mkey_seg           mkc;
-	struct mlx5_mtt                inline_mtts[0];
+	union {
+		struct mlx5_mtt        inline_mtts[0];
+		struct mlx5_klm	       inline_klms[0];
+	};
 };
 
 extern const char mlx5e_self_tests[][ETH_GSTRING_LEN];
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +3 -0
index 055c3bc23733..529745aac7e8 100644
@@ -41,6 +41,9 @@ enum mlx5e_icosq_wqe_type {
 	MLX5E_ICOSQ_WQE_SET_PSV_TLS,
 	MLX5E_ICOSQ_WQE_GET_PSV_TLS,
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	MLX5E_ICOSQ_WQE_UMR_NVME_TCP,
+#endif
 };
 
 /* General */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +116 -0

128 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h new file +12 -0
new file mode 100644
index 000000000000..329e114d6571
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+#ifndef __MLX5E_NVMEOTCP_UTILS_H__
+#define __MLX5E_NVMEOTCP_UTILS_H__
+
+#include "en.h"
+
+#define MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi) \
+	((struct mlx5e_umr_wqe *)\
+	 mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_umr_wqe)))
+
+#endif /* __MLX5E_NVMEOTCP_UTILS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +10 -2
index 3c65fd0bcf31..9d821facbca4 100644
@@ -615,16 +615,20 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
 		ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc);
 		wi = &sq->db.wqe_info[ci];
 		sqcc += wi->num_wqebbs;
-#ifdef CONFIG_MLX5_EN_TLS
 		switch (wi->wqe_type) {
+#ifdef CONFIG_MLX5_EN_TLS
 		case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
 			mlx5e_ktls_handle_ctx_completion(wi);
 			break;
 		case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
 			mlx5e_ktls_handle_get_psv_completion(wi, sq);
 			break;
-		}
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
+			break;
+#endif
+		}
 	}
 	sq->cc = sqcc;
 }
@@ -694,6 +698,10 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 			case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
 				mlx5e_ktls_handle_get_psv_completion(wi, sq);
 				break;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
+				break;
 #endif
 			default:
 				netdev_WARN_ONCE(cq->netdev,
Signature · 1 line
2.24.1
author reviewer
#33 lore raw

PATCH v5 15/36 net/mlx5e: NVMEoTCP queue init/teardown

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

When nvme-tcp establishes new connections, we allocate a hardware context to offload operations for this queue:
- Use a separate TIR to identify the queue and maintain the HW context
- Use a separate ICOSQ for maintain the HW context
- Use a separate tag buffer for buffer registration
- Maintain static and progress HW contexts by posting the proper WQEs at creation time, or upon resync

Queue teardown will free the corresponding contexts.

---
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   6 +
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 663 +++++++++++++++++-
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    |   4 +
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |  68 ++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |   7 +
 5 files changed, 723 insertions(+), 25 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +6 -0
index 529745aac7e8..497c49f28d8a 100644
@@ -43,6 +43,7 @@ enum mlx5e_icosq_wqe_type {
 #endif
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 	MLX5E_ICOSQ_WQE_UMR_NVME_TCP,
+	MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP,
 #endif
 };
 
@@ -185,6 +186,11 @@ struct mlx5e_icosq_wqe_info {
 		struct {
 			struct mlx5e_ktls_rx_resync_buf *buf;
 		} tls_get_params;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+		struct {
+			struct mlx5e_nvmeotcp_queue *queue;
+		} nvmeotcp_q;
 #endif
 	};
 };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +638 -25

777 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +4 -0
index b4a27a03578e..20141010817d 100644
@@ -102,6 +102,10 @@ int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv);
 int set_feature_nvme_tcp(struct net_device *netdev, bool enable);
 int set_feature_nvme_tcp_crc(struct net_device *netdev, bool enable);
 void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);
+struct mlx5e_nvmeotcp_queue *
+mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id);
+void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue);
+void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi);
 int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv);
 void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
 #else
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h +68 -0

77 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +7 -0
index 9d821facbca4..bc7b19974ed9 100644
@@ -47,6 +47,7 @@
 #include "fpga/ipsec.h"
 #include "en_accel/ipsec_rxtx.h"
 #include "en_accel/tls_rxtx.h"
+#include "en_accel/nvmeotcp.h"
 #include "en/xdp.h"
 #include "en/xsk/rx.h"
 #include "en/health.h"
@@ -627,6 +628,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
 			break;
+		case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
+			mlx5e_nvmeotcp_ctx_comp(wi);
+			break;
 #endif
 		}
 	}
@@ -702,6 +706,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
 				break;
+			case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
+				mlx5e_nvmeotcp_ctx_comp(wi);
+				break;
 #endif
 			default:
 				netdev_WARN_ONCE(cq->netdev,
Signature · 1 line
2.24.1
author reviewer
#34 lore raw

PATCH v5 16/36 net/mlx5e: NVMEoTCP async ddp invalidation

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

Teardown ddp contexts asynchronously by posting a WQE, and calling back to nvme-tcp when the corresponding CQE is received.

---
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |  4 ++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 66 ++++++++++++++++---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    |  1 +
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  6 ++
 4 files changed, 69 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +4 -0
index 497c49f28d8a..f0190ee6e42c 100644
@@ -43,6 +43,7 @@ enum mlx5e_icosq_wqe_type {
 #endif
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 	MLX5E_ICOSQ_WQE_UMR_NVME_TCP,
+	MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE,
 	MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP,
 #endif
 };
@@ -191,6 +192,9 @@ struct mlx5e_icosq_wqe_info {
 		struct {
 			struct mlx5e_nvmeotcp_queue *queue;
 		} nvmeotcp_q;
+		struct {
+			struct nvmeotcp_queue_entry *entry;
+		} nvmeotcp_qe;
 #endif
 	};
 };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +58 -8

131 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +1 -0
index 20141010817d..b9642e130b97 100644
@@ -105,6 +105,7 @@ void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);
 struct mlx5e_nvmeotcp_queue *
 mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id);
 void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue);
+void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi);
 void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi);
 int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv);
 void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +6 -0
index bc7b19974ed9..e92dd4666955 100644
@@ -628,6 +628,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
 			break;
+		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE:
+			mlx5e_nvmeotcp_ddp_inv_done(wi);
+			break;
 		case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
 			mlx5e_nvmeotcp_ctx_comp(wi);
 			break;
@@ -706,6 +709,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 #ifdef CONFIG_MLX5_EN_NVMEOTCP
 			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
 				break;
+			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE:
+				mlx5e_nvmeotcp_ddp_inv_done(wi);
+				break;
 			case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
 				mlx5e_nvmeotcp_ctx_comp(wi);
 				break;
Signature · 1 line
2.24.1
author reviewer
#35 lore raw

PATCH v5 17/36 net/mlx5e: NVMEoTCP ddp setup and resync

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

NVMEoTCP offload uses buffer registration for every NVME request to perform direct data placement, The registration is done via KLM UMR WQE's. The driver resync handler advertise the software resync response via static params WQE.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +29 -0
index 864b080cc8fc..4fdfbe4468ee 100644
@@ -762,6 +762,30 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
 			 struct sock *sk,
 			 struct ulp_ddp_io *ddp)
 {
+	struct mlx5e_priv *priv = netdev_priv(netdev);
+	struct scatterlist *sg = ddp->sg_table.sgl;
+	struct mlx5e_nvmeotcp_queue *queue;
+	struct mlx5_core_dev *mdev;
+	int i, size = 0, count = 0;
+
+	queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+
+	mdev = queue->priv->mdev;
+	count = dma_map_sg(mdev->device, ddp->sg_table.sgl, ddp->nents,
+			   DMA_FROM_DEVICE);
+
+	if (WARN_ON(count > mlx5e_get_max_sgl(mdev)))
+		return -ENOSPC;
+
+	for (i = 0; i < count; i++)
+		size += sg[i].length;
+
+	queue->ccid_table[ddp->command_id].size = size;
+	queue->ccid_table[ddp->command_id].ddp = ddp;
+	queue->ccid_table[ddp->command_id].sgl = sg;
+	queue->ccid_table[ddp->command_id].ccid_gen++;
+	queue->ccid_table[ddp->command_id].sgl_length = count;
+
 	return 0;
 }
 
@@ -819,6 +843,11 @@ static void
 mlx5e_nvmeotcp_dev_resync(struct net_device *netdev,
 			  struct sock *sk, u32 seq)
 {
+	struct mlx5e_nvmeotcp_queue *queue =
+		container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+
+	queue->after_resync_cqe = 1;
+	mlx5e_nvmeotcp_rx_post_static_params_wqe(queue, seq);
 }
 
 static const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = {
Signature · 1 line
2.24.1
author reviewer
#36 lore raw

PATCH v5 18/36 net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

This patch implements the data-path for direct data placement (DDP) and DDGST offloads. NVMEoTCP DDP constructs an SKB from each CQE, while pointing at NVME destination buffers. In turn, this enables the offload, as the NVMe-TCP layer will skip the copy when src == dst.

Additionally, this patch adds support for DDGST (CRC32) offload. HW will report DDGST offload only if it has not encountered an error in the received packet. We pass this indication in skb->ddp_crc up the stack to NVMe-TCP to skip computing the DDGST if all corresponding SKBs were verified by HW.

This patch also handles context resynchronization requests made by NIC HW. The resync request is passed to the NVMe-TCP layer to be handled at a later point in time.

Finally, we also use the skb->ddp_crc bit to avoid skb_condense. This is critical as every SKB that uses DDP has a hole that fits perfectly with skb_condense's policy, but filling this hole is counter-productive as the data there already resides in its destination buffer.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |   1 +
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        | 248 ++++++++++++++++++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.h        |  43 +++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  41 ++-
 7 files changed, 330 insertions(+), 7 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
index 0ae9e5e38ec7..992b396907ae 100644
@@ -102,4 +102,4 @@ mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o
 #
 mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/hw_table.o sf/devlink.o
 
-mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
index d88ecd04d5da..c87f32492ea4 100644
@@ -589,6 +589,7 @@ struct mlx5e_rq;
 typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, struct mlx5_cqe64*);
 typedef struct sk_buff *
 (*mlx5e_fp_skb_from_cqe_mpwrq)(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
+			       struct mlx5_cqe64 *cqe,
 			       u16 cqe_bcnt, u32 head_offset, u32 page_idx);
 typedef struct sk_buff *
 (*mlx5e_fp_skb_from_cqe)(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
index 8e7b877d8a12..9a6fbd1b1c34 100644
@@ -25,6 +25,7 @@ static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, void *data,
 
 struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 						    struct mlx5e_mpw_info *wi,
+						    struct mlx5_cqe64 *cqe,
 						    u16 cqe_bcnt,
 						    u32 head_offset,
 						    u32 page_idx)
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +1 -0
index 7f88ccf67fdd..112c5b3ec165 100644
@@ -11,6 +11,7 @@
 
 struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 						    struct mlx5e_mpw_info *wi,
+						    struct mlx5_cqe64 *cqe,
 						    u16 cqe_bcnt,
 						    u32 head_offset,
 						    u32 page_idx);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c new file +248 -0

248 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h new file +43 -0

43 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +35 -6

125 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#37 lore raw

PATCH v5 19/36 net/mlx5e: NVMEoTCP statistics

In reply to Boris Pismenny

From: Ben Ben-Ishay <[email protected]>

NVMEoTCP offload statistics includes both control and data path
statistic: counters for ndo, offloaded packets/bytes, dropped packets and resync operation.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 23 +++++++++++-
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        | 16 ++++++++
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 37 +++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/en_stats.h    | 24 ++++++++++++
 4 files changed, 98 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +21 -2
index 4fdfbe4468ee..7f6607cac9fb 100644
@@ -667,6 +667,11 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	struct mlx5_core_dev *mdev = priv->mdev;
 	struct mlx5e_nvmeotcp_queue *queue;
 	int max_wqe_sz_cap, queue_id, err;
+	struct mlx5e_rq_stats *stats;
+	u32 channel_ix;
+
+	channel_ix = mlx5e_get_channel_ix_from_io_cpu(priv, config->io_cpu);
+	stats = &priv->channel_stats[channel_ix].rq;
 
 	if (tconfig->type != ULP_DDP_NVME) {
 		err = -EOPNOTSUPP;
@@ -694,8 +699,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	queue->id = queue_id;
 	queue->dgst = config->dgst;
 	queue->pda = config->cpda;
-	queue->channel_ix = mlx5e_get_channel_ix_from_io_cpu(priv,
-							     config->io_cpu);
+	queue->channel_ix = channel_ix;
 	queue->size = config->queue_size;
 	max_wqe_sz_cap  = min_t(int, MAX_DS_VALUE * MLX5_SEND_WQE_DS,
 				MLX5_CAP_GEN(mdev, max_wqe_sz_sq) << OCTWORD_SHIFT);
@@ -715,6 +719,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	if (err)
 		goto destroy_rx;
 
+	stats->nvmeotcp_queue_init++;
 	write_lock_bh(&sk->sk_callback_lock);
 	ulp_ddp_set_ctx(sk, queue);
 	write_unlock_bh(&sk->sk_callback_lock);
@@ -729,6 +734,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 free_queue:
 	kfree(queue);
 out:
+	stats->nvmeotcp_queue_init_fail++;
 	return err;
 }
 
@@ -739,11 +745,15 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
 	struct mlx5e_priv *priv = netdev_priv(netdev);
 	struct mlx5_core_dev *mdev = priv->mdev;
 	struct mlx5e_nvmeotcp_queue *queue;
+	struct mlx5e_rq_stats *stats;
 
 	queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
 
 	napi_synchronize(&priv->channels.c[queue->channel_ix]->napi);
 
+	stats = &priv->channel_stats[queue->channel_ix].rq;
+	stats->nvmeotcp_queue_teardown++;
+
 	WARN_ON(refcount_read(&queue->ref_count) != 1);
 	if (queue->zerocopy | queue->crc_rx)
 		mlx5e_nvmeotcp_destroy_rx(queue, mdev, queue->zerocopy);
@@ -765,6 +775,7 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
 	struct mlx5e_priv *priv = netdev_priv(netdev);
 	struct scatterlist *sg = ddp->sg_table.sgl;
 	struct mlx5e_nvmeotcp_queue *queue;
+	struct mlx5e_rq_stats *stats;
 	struct mlx5_core_dev *mdev;
 	int i, size = 0, count = 0;
 
@@ -786,6 +797,11 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
 	queue->ccid_table[ddp->command_id].ccid_gen++;
 	queue->ccid_table[ddp->command_id].sgl_length = count;
 
+	stats = &priv->channel_stats[queue->channel_ix].rq;
+	stats->nvmeotcp_ddp_setup++;
+	if (unlikely(mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, count)))
+		stats->nvmeotcp_ddp_setup_fail++;
+
 	return 0;
 }
 
@@ -826,6 +842,7 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
 	struct mlx5e_nvmeotcp_queue *queue;
 	struct mlx5e_priv *priv = netdev_priv(netdev);
 	struct nvmeotcp_queue_entry *q_entry;
+	struct mlx5e_rq_stats *stats;
 
 	queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
 	q_entry  = &queue->ccid_table[ddp->command_id];
@@ -835,6 +852,8 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
 	q_entry->queue = queue;
 
 	mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, 0);
+	stats = &priv->channel_stats[queue->channel_ix].rq;
+	stats->nvmeotcp_ddp_teardown++;
 
 	return 0;
 }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c +16 -0
index 31586f574fc0..d4ac914e2a5b 100644
@@ -10,12 +10,16 @@ static void nvmeotcp_update_resync(struct mlx5e_nvmeotcp_queue *queue,
 				   struct mlx5e_cqe128 *cqe128)
 {
 	const struct ulp_ddp_ulp_ops *ulp_ops;
+	struct mlx5e_rq_stats *stats;
 	u32 seq;
 
 	seq = be32_to_cpu(cqe128->resync_tcp_sn);
 	ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops;
 	if (ulp_ops && ulp_ops->resync_request)
 		ulp_ops->resync_request(queue->sk, seq, ULP_DDP_RESYNC_REQ);
+
+	stats = queue->priv->channels.c[queue->channel_ix]->rq.stats;
+	stats->nvmeotcp_resync++;
 }
 
 static void mlx5e_nvmeotcp_advance_sgl_iter(struct mlx5e_nvmeotcp_queue *queue)
@@ -50,10 +54,13 @@ mlx5_nvmeotcp_add_tail_nonlinear(struct mlx5e_nvmeotcp_queue *queue,
 				 int org_nr_frags, int frag_index)
 {
 	struct mlx5e_priv *priv = queue->priv;
+	struct mlx5e_rq_stats *stats;
 
 	while (org_nr_frags != frag_index) {
 		if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) {
 			dev_kfree_skb_any(skb);
+			stats = priv->channels.c[queue->channel_ix]->rq.stats;
+			stats->nvmeotcp_drop++;
 			return NULL;
 		}
 		skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
@@ -72,9 +79,12 @@ mlx5_nvmeotcp_add_tail(struct mlx5e_nvmeotcp_queue *queue, struct sk_buff *skb,
 		       int offset, int len)
 {
 	struct mlx5e_priv *priv = queue->priv;
+	struct mlx5e_rq_stats *stats;
 
 	if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) {
 		dev_kfree_skb_any(skb);
+		stats = priv->channels.c[queue->channel_ix]->rq.stats;
+		stats->nvmeotcp_drop++;
 		return NULL;
 	}
 	skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
@@ -135,6 +145,7 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 	skb_frag_t org_frags[MAX_SKB_FRAGS];
 	struct mlx5e_nvmeotcp_queue *queue;
 	struct nvmeotcp_queue_entry *nqe;
+	struct mlx5e_rq_stats *stats;
 	int org_nr_frags, frag_index;
 	struct mlx5e_cqe128 *cqe128;
 	u32 queue_id;
@@ -172,6 +183,8 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 		return skb;
 	}
 
+	stats = priv->channels.c[queue->channel_ix]->rq.stats;
+
 	/* cc ddp from cqe */
 	ccid = be16_to_cpu(cqe128->ccid);
 	ccoff = be32_to_cpu(cqe128->ccoff);
@@ -214,6 +227,7 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 	while (to_copy < cclen) {
 		if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) {
 			dev_kfree_skb_any(skb);
+			stats->nvmeotcp_drop++;
 			mlx5e_nvmeotcp_put_queue(queue);
 			return NULL;
 		}
@@ -243,6 +257,8 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
 							       frag_index);
 	}
 
+	stats->nvmeotcp_offload_packets++;
+	stats->nvmeotcp_offload_bytes += cclen;
 	mlx5e_nvmeotcp_put_queue(queue);
 	return skb;
 }
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +37 -0
index e4f5b6395148..3d95e46422e5 100644
@@ -34,6 +34,7 @@
 #include "en.h"
 #include "en_accel/tls.h"
 #include "en_accel/en_accel.h"
+#include "en_accel/nvmeotcp.h"
 
 static unsigned int stats_grps_num(struct mlx5e_priv *priv)
 {
@@ -187,6 +188,18 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_retry) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_skip) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) },
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init_fail) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_teardown) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_drop) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_resync) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_bytes) },
 #endif
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
@@ -349,6 +362,18 @@ static void mlx5e_stats_grp_sw_update_stats_rq_stats(struct mlx5e_sw_stats *s,
 	s->rx_tls_resync_res_skip     += rq_stats->tls_resync_res_skip;
 	s->rx_tls_err                 += rq_stats->tls_err;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	s->rx_nvmeotcp_queue_init      += rq_stats->nvmeotcp_queue_init;
+	s->rx_nvmeotcp_queue_init_fail += rq_stats->nvmeotcp_queue_init_fail;
+	s->rx_nvmeotcp_queue_teardown  += rq_stats->nvmeotcp_queue_teardown;
+	s->rx_nvmeotcp_ddp_setup       += rq_stats->nvmeotcp_ddp_setup;
+	s->rx_nvmeotcp_ddp_setup_fail  += rq_stats->nvmeotcp_ddp_setup_fail;
+	s->rx_nvmeotcp_ddp_teardown    += rq_stats->nvmeotcp_ddp_teardown;
+	s->rx_nvmeotcp_drop            += rq_stats->nvmeotcp_drop;
+	s->rx_nvmeotcp_resync          += rq_stats->nvmeotcp_resync;
+	s->rx_nvmeotcp_offload_packets += rq_stats->nvmeotcp_offload_packets;
+	s->rx_nvmeotcp_offload_bytes   += rq_stats->nvmeotcp_offload_bytes;
+#endif
 }
 
 static void mlx5e_stats_grp_sw_update_stats_ch_stats(struct mlx5e_sw_stats *s,
@@ -1788,6 +1813,18 @@ static const struct counter_desc rq_stats_desc[] = {
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_resync_res_skip) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) },
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init_fail) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_teardown) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_drop) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_resync) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_bytes) },
+#endif
 };
 
 static const struct counter_desc sq_stats_desc[] = {
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +24 -0
index 139e59f30db0..e645ee83de97 100644
@@ -194,6 +194,18 @@ struct mlx5e_sw_stats {
 	u64 rx_congst_umr;
 	u64 rx_arfs_err;
 	u64 rx_recover;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	u64 rx_nvmeotcp_queue_init;
+	u64 rx_nvmeotcp_queue_init_fail;
+	u64 rx_nvmeotcp_queue_teardown;
+	u64 rx_nvmeotcp_ddp_setup;
+	u64 rx_nvmeotcp_ddp_setup_fail;
+	u64 rx_nvmeotcp_ddp_teardown;
+	u64 rx_nvmeotcp_drop;
+	u64 rx_nvmeotcp_resync;
+	u64 rx_nvmeotcp_offload_packets;
+	u64 rx_nvmeotcp_offload_bytes;
+#endif
 	u64 ch_events;
 	u64 ch_poll;
 	u64 ch_arm;
@@ -354,6 +366,18 @@ struct mlx5e_rq_stats {
 	u64 tls_resync_res_skip;
 	u64 tls_err;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	u64 nvmeotcp_queue_init;
+	u64 nvmeotcp_queue_init_fail;
+	u64 nvmeotcp_queue_teardown;
+	u64 nvmeotcp_ddp_setup;
+	u64 nvmeotcp_ddp_setup_fail;
+	u64 nvmeotcp_ddp_teardown;
+	u64 nvmeotcp_drop;
+	u64 nvmeotcp_resync;
+	u64 nvmeotcp_offload_packets;
+	u64 nvmeotcp_offload_bytes;
+#endif
 };
 
 struct mlx5e_sq_stats {
Signature · 1 line
2.24.1
author reviewer
#38 lore raw

PATCH v5 20/36 Documentation: add ULP DDP offload documentation

In reply to Boris Pismenny

From: Boris Pismenny <[email protected]>

---
 Documentation/networking/index.rst           |   1 +
 Documentation/networking/ulp-ddp-offload.rst | 415 +++++++++++++++++++
 2 files changed, 416 insertions(+)
 create mode 100644 Documentation/networking/ulp-ddp-offload.rst
Documentation/networking/index.rst +1 -0
index e9ce55992aa9..87c08683e006 100644
@@ -100,6 +100,7 @@ Contents:
    sysfs-tagging
    tc-actions-env-rules
    tcp-thin
+   ulp-ddp-offload
    team
    timestamping
    tipc
Documentation/networking/ulp-ddp-offload.rst new file +415 -0

415 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#39 lore raw

PATCH v5 21/36 net: drop ULP DDP HW offload feature if no CSUM offload feature

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

---
 net/core/dev.c | 2 ++
 1 file changed, 2 insertions(+)
net/core/dev.c +2 -0
index c253c2aafe97..1c8fcee2e9f9 100644
@@ -9927,6 +9927,8 @@ static netdev_features_t netdev_fix_features(struct net_device *dev,
 		if (!ip_csum && !hw_csum) {
 			netdev_dbg(dev, "Dropping TLS TX HW offload feature since no CSUM feature.\n");
 			features &= ~NETIF_F_HW_TLS_TX;
+			netdev_dbg(dev, "Dropping ULP DDP HW offload feature since no CSUM feature.\n");
+			features &= ~NETIF_F_HW_ULP_DDP;
 		}
 	}
 
Signature · 1 line
2.24.1
author reviewer
#40 lore raw

PATCH v5 22/36 net: Add ulp_ddp_pdu_info struct

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

This struct is mapping between pdu's pages to TCP sequence number.

The use case for this mapping is in tx offload,
when the NIC needs to send a retransmitted packet. The NIC, in this case, might need the complete pdu, which is sent before (i.e for computing the CRC for this pdu).

Using this mapping, the SW can send the NIC enough context to offload the packet.

---
 include/net/ulp_ddp.h |  55 +++++++++++++-
 net/core/Makefile     |   1 +
 net/core/ulp_ddp.c    | 166 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 219 insertions(+), 3 deletions(-)
 create mode 100644 net/core/ulp_ddp.c
include/net/ulp_ddp.h +52 -3
index 1a0b464ff40b..8f48fc121c3a 100644
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-3.0
  *
  * ulp_ddp.h
  *	Author:	Boris Pismenny <[email protected]>
@@ -10,6 +10,7 @@
 #include <linux/netdevice.h>
 #include <net/inet_connection_sock.h>
 #include <net/sock.h>
+#include <net/tcp.h>
 
 /* limits returned by the offload driver, zero means don't care */
 struct ulp_ddp_limits {
@@ -67,6 +68,26 @@ struct ulp_ddp_io {
 	struct scatterlist	first_sgl[SG_CHUNK_SIZE];
 };
 
+/**
+ * struct ulp_ddp_pdu_info - pdu info for tcp ddp crc Tx offload.
+ *
+ * @end_seq:	tcp seq of the last byte in the pdu.
+ * @start_seq:	tcp seq of the first byte in the pdu.
+ * @data_len:	pdu data size (in bytes).
+ * @hdr_len:	the size (in bytes) of the pdu header.
+ * @hdr:	pdu header.
+ * @req:	the ulp request for the original pdu.
+ */
+struct ulp_ddp_pdu_info {
+	struct list_head list;
+	u32		end_seq;
+	u32		start_seq;
+	u32		data_len;
+	u32		hdr_len;
+	void		*hdr;
+	struct request	*req;
+};
+
 /* struct ulp_ddp_dev_ops - operations used by an upper layer protocol to configure ddp offload
  *
  * @ulp_ddp_limits:    limit the number of scatter gather entries per IO.
@@ -113,10 +134,25 @@ struct ulp_ddp_ulp_ops {
 /**
  * struct ulp_ddp_ctx - Generic ulp ddp context: device driver per queue contexts must
  * use this as the first member.
+ *
+ * @netdev:		the coresponding netdev for this tcp ddp.
+ * @ddgst_len:		data digest len in bytes.
+ * @expected_seq:	indicates for next tcp seq.
+ * @open_info:		the current pdu_info.
+ * @pdu_hint:		hint for ulp_ddp_get_pdu_info.
+ * @info_list:		list of the mapped pdu_infos.
+ * @info_lock:		lock for info_list.
  */
 struct ulp_ddp_ctx {
-	enum ulp_ddp_type    type;
-	unsigned char        buf[];
+	enum ulp_ddp_type	type;
+	struct net_device	*netdev;
+	int			ddgst_len;
+	u32			expected_seq;
+	struct ulp_ddp_pdu_info *open_info;
+	struct ulp_ddp_pdu_info *pdu_hint;
+	struct list_head        info_list;
+	spinlock_t              info_lock;
+	unsigned char           buf[];
 };
 
 static inline struct ulp_ddp_ctx *ulp_ddp_get_ctx(const struct sock *sk)
@@ -133,4 +169,17 @@ static inline void ulp_ddp_set_ctx(struct sock *sk, void *ctx)
 	rcu_assign_pointer(icsk->icsk_ulp_ddp_data, ctx);
 }
 
+static inline void ulp_ddp_destroy_info(struct ulp_ddp_pdu_info *info)
+{
+	kfree(info);
+}
+
+void ulp_ddp_ack_handle(struct sock *sk, u32 acked_seq);
+int ulp_ddp_init_tx_offload(struct sock *sk);
+void ulp_ddp_release_tx_offload(struct sock *sk);
+int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr,
+			 u32 hdr_len, u32 data_len, struct request *req);
+void ulp_ddp_close_pdu_info(struct sock *sk);
+bool ulp_ddp_need_map(struct sock *sk);
+struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq);
 #endif //_ULP_DDP_H
net/core/Makefile +1 -0
index f7f16650fe9e..b7c1618944df 100644
@@ -14,6 +14,7 @@ obj-y		     += dev.o dev_addr_lists.o dst.o netevent.o \
 			fib_notifier.o xdp.o flow_offload.o
 
 obj-y += net-sysfs.o
+obj-$(CONFIG_ULP_DDP) += ulp_ddp.o
 obj-$(CONFIG_PAGE_POOL) += page_pool.o
 obj-$(CONFIG_PROC_FS) += net-procfs.o
 obj-$(CONFIG_NET_PKTGEN) += pktgen.o
net/core/ulp_ddp.c new file +166 -0

166 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#42 lore raw

PATCH v5 23/36 net: Add to ulp_ddp support for fallback flow

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

Add ddp_ddgest_falback(), and ddp_get_pdu_info function to ulp.

During DDP CRC Tx offload, the HW is responsible for calculate the crc, and therefore the SW not calculates it.

If the HW changes for some reason,
the SW should fallback from the offload and calculate the crc. This is checking in the ulp_ddp_validate_skb and if need fallback it do it.

---
 include/net/ulp_ddp.h |  7 +++++
 net/core/ulp_ddp.c    | 69 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
include/net/ulp_ddp.h +7 -0
index 8f48fc121c3a..40bfcfe94cef 100644
@@ -77,6 +77,7 @@ struct ulp_ddp_io {
  * @hdr_len:	the size (in bytes) of the pdu header.
  * @hdr:	pdu header.
  * @req:	the ulp request for the original pdu.
+ * @ddgst:	pdu data digest.
  */
 struct ulp_ddp_pdu_info {
 	struct list_head list;
@@ -86,6 +87,7 @@ struct ulp_ddp_pdu_info {
 	u32		hdr_len;
 	void		*hdr;
 	struct request	*req;
+	__le32          ddgst;
 };
 
 /* struct ulp_ddp_dev_ops - operations used by an upper layer protocol to configure ddp offload
@@ -129,6 +131,8 @@ struct ulp_ddp_ulp_ops {
 	bool (*resync_request)(struct sock *sk, u32 seq, u32 flags);
 	/* NIC driver informs the ulp that ddp teardown is done - used for async completions*/
 	void (*ddp_teardown_done)(void *ddp_ctx);
+	/* NIC request ulp to calculate the ddgst and store it in pdu_info->ddgst */
+	void (*ddp_ddgst_fallback)(struct ulp_ddp_pdu_info *pdu_info);
 };
 
 /**
@@ -182,4 +186,7 @@ int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr,
 void ulp_ddp_close_pdu_info(struct sock *sk);
 bool ulp_ddp_need_map(struct sock *sk);
 struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq);
+struct sk_buff *ulp_ddp_validate_xmit_skb(struct sock *sk,
+					  struct net_device *dev,
+					  struct sk_buff *skb);
 #endif //_ULP_DDP_H
net/core/ulp_ddp.c +69 -0

72 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#44 lore raw

PATCH v5 24/36 net: Add MSG_DDP_CRC flag

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

if the msg sent with this flag, turn up skb->ddp_crc bit.

---
 include/linux/socket.h | 1 +
 include/net/sock.h     | 6 ++++++
 net/core/sock.c        | 7 +++++++
 net/ipv4/tcp.c         | 6 ++++++
 4 files changed, 20 insertions(+)
include/linux/socket.h +1 -0
index 0d8e3dcb7f88..640ec8535f43 100644
@@ -310,6 +310,7 @@ struct ucred {
 					  * plain text and require encryption
 					  */
 
+#define MSG_DDP_CRC  0x200000		/* Skb pdu need crc offload */
 #define MSG_ZEROCOPY	0x4000000	/* Use user data in kernel path */
 #define MSG_FASTOPEN	0x20000000	/* Send data in TCP SYN */
 #define MSG_CMSG_CLOEXEC 0x40000000	/* Set close_on_exec for file
include/net/sock.h +6 -0
index 5fa2fd192d18..84141fdc3b80 100644
@@ -2656,6 +2656,12 @@ static inline struct sk_buff *sk_validate_xmit_skb(struct sk_buff *skb,
 		pr_warn_ratelimited("unencrypted skb with no associated socket - dropping\n");
 		kfree_skb(skb);
 		skb = NULL;
+#endif
+#ifdef CONFIG_ULP_DDP
+	} else if (unlikely(skb->ddp_crc)) {
+		pr_warn_ratelimited("crc-offload skb with no associated socket - dropping\n");
+		kfree_skb(skb);
+		skb = NULL;
 #endif
 	}
 #endif
net/core/sock.c +7 -0
index ba1c0f75cd45..616ffc523b5d 100644
@@ -2160,6 +2160,13 @@ static bool can_skb_orphan_partial(const struct sk_buff *skb)
 	 */
 	if (skb->decrypted)
 		return false;
+#endif
+#ifdef CONFIG_ULP_DDP
+	/* Drivers depend on in-order delivery for crc offload,
+	 * partial orphan breaks out-of-order-OK logic.
+	 */
+	if (skb->ddp_crc)
+		return false;
 #endif
 	return (skb->destructor == sock_wfree ||
 		(IS_ENABLED(CONFIG_INET) && skb->destructor == tcp_wfree));
net/ipv4/tcp.c +6 -0
index d5ab5f243640..36c445ed8a30 100644
@@ -984,6 +984,9 @@ struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags,
 
 #ifdef CONFIG_TLS_DEVICE
 		skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED);
+#endif
+#ifdef CONFIG_ULP_DDP
+		skb->ddp_crc = !!(flags & MSG_DDP_CRC);
 #endif
 		skb_entail(sk, skb);
 		copy = size_goal;
@@ -1311,6 +1314,9 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 			if (!skb)
 				goto wait_for_space;
 
+#ifdef CONFIG_ULP_DDP
+			skb->ddp_crc = !!(flags & MSG_DDP_CRC);
+#endif
 			process_backlog++;
 			skb->ip_summed = CHECKSUM_PARTIAL;
 
Signature · 1 line
2.24.1
author reviewer
#46 lore raw

PATCH v5 25/36 nvme-tcp: TX DDGST offload

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

This patch add support for TX DDGST offload.

Enable tx side of DDGST offload when supported.

if supported, NVMEoTCP will:
1. Stop compute the DDGST on transmitted pdus.
2. send dummy digest (only zeros).

---
 drivers/nvme/host/tcp.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
drivers/nvme/host/tcp.c +27 -6
index b338cd2d9f65..b2a4316eddce 100644
@@ -70,6 +70,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_POLLING	= 2,
 	NVME_TCP_Q_OFF_DDP	= 3,
 	NVME_TCP_Q_OFF_DDGST_RX = 4,
+	NVME_TCP_Q_OFF_DDGST_TX = 5,
 };
 
 enum nvme_tcp_recv_state {
@@ -372,6 +373,7 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
 	if (netdev->features & NETIF_F_HW_ULP_DDP) {
 		set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
 		set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
+		set_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags);
 	}
 
 	return ret;
@@ -388,6 +390,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
 
 	clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
 	clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
+	clear_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags);
 
 	netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk);
 
@@ -1269,6 +1272,7 @@ static void nvme_tcp_fail_request(struct nvme_tcp_request *req)
 static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 {
 	struct nvme_tcp_queue *queue = req->queue;
+	bool is_offload = test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags);
 
 	while (true) {
 		struct page *page = nvme_tcp_req_cur_page(req);
@@ -1277,6 +1281,9 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 		bool last = nvme_tcp_pdu_last_send(req, len);
 		int ret, flags = MSG_DONTWAIT;
 
+		if (is_offload && queue->data_digest)
+			flags |= MSG_DDP_CRC;
+
 		if (last && !queue->data_digest && !nvme_tcp_queue_more(queue))
 			flags |= MSG_EOR;
 		else
@@ -1292,15 +1299,19 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 		if (ret <= 0)
 			return ret;
 
-		if (queue->data_digest)
+		if (queue->data_digest && !is_offload)
 			nvme_tcp_ddgst_update(queue->snd_hash, page,
 					offset, ret);
 
 		/* fully successful last write*/
 		if (last && ret == len) {
 			if (queue->data_digest) {
-				nvme_tcp_ddgst_final(queue->snd_hash,
-					&req->ddgst);
+				if (!is_offload)
+					nvme_tcp_ddgst_final(queue->snd_hash,
+						&req->ddgst);
+				else
+					req->ddgst = 0;
+
 				req->state = NVME_TCP_SEND_DDGST;
 				req->offset = 0;
 			} else {
@@ -1324,6 +1335,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	int flags = MSG_DONTWAIT;
 	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest)
+		flags |= MSG_DDP_CRC;
+
 	if (inline_data || nvme_tcp_queue_more(queue))
 		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
 	else
@@ -1357,18 +1371,21 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 
 static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)
 {
+	int flags = MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST;
 	struct nvme_tcp_queue *queue = req->queue;
 	struct nvme_tcp_data_pdu *pdu = req->pdu;
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) - req->offset + hdgst;
 	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest)
+		flags |= MSG_DDP_CRC;
+
 	if (queue->hdr_digest && !req->offset)
 		nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu));
 
 	ret = kernel_sendpage(queue->sock, virt_to_page(pdu),
-			offset_in_page(pdu) + req->offset, len,
-			MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST);
+			offset_in_page(pdu) + req->offset, len, flags);
 	if (unlikely(ret <= 0))
 		return ret;
 
@@ -1399,6 +1416,9 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 	else
 		msg.msg_flags |= MSG_EOR;
 
+	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags))
+		msg.msg_flags |= MSG_DDP_CRC;
+
 	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
 	if (unlikely(ret <= 0))
 		return ret;
@@ -1908,7 +1928,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
 	cancel_work_sync(&queue->io_work);
 
 	if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) ||
-	    test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
+	    test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags) ||
+	    test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags))
 		nvme_tcp_unoffload_socket(queue);
 }
 
Signature · 1 line
2.24.1
author reviewer
#47 lore raw

PATCH v5 26/36 nvme-tcp: Mapping between Tx NVMEoTCP pdu and TCP sequence

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

This commit maintains a mapping from TCP sequence number to NVMEoTCP pdus, for DDGST tx offload using the ULP_DDP API.
When send a pdu it save the req in ulp_ddp_pdu_info struct.

This mapping is used:
1. When packet is retransmitted, If this packet contain NVMEoTCP DDGST, The NIC might needs all the pdu again for computing the DDGST.
2. If packet is offloaded but will not go to the offloaded netdev, Then SW will need to be able to fallback and compute the DDGST.

Add founction nvme_tcp_ddgest_fallback(pdu_info):
caclulate the data digest for ulp_ddp_pdu_info (requested by the netdev).

---
 drivers/nvme/host/tcp.c | 93 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 91 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +91 -2
index b2a4316eddce..e030d1baa6bb 100644
@@ -154,6 +154,55 @@ static const struct blk_mq_ops nvme_tcp_mq_ops;
 static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
 static int nvme_tcp_try_send(struct nvme_tcp_queue *queue);
 
+#ifdef CONFIG_ULP_DDP
+static int nvme_tcp_map_pdu_info(struct nvme_tcp_queue *queue,
+				  size_t sent_size,
+				  u32 pdu_len, u32 data_len)
+{
+	u32 start_seq = tcp_sk(queue->sock->sk)->write_seq - sent_size;
+	struct nvme_tcp_request *req = queue->request;
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	return ulp_ddp_map_pdu_info(queue->sock->sk, start_seq, req->pdu,
+				    pdu_len, data_len, rq);
+}
+
+static void nvme_tcp_close_pdu_info(struct nvme_tcp_queue *queue)
+{
+	if (queue->data_digest &&
+	    test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags))
+		ulp_ddp_close_pdu_info(queue->sock->sk);
+}
+
+bool nvme_tcp_need_map(struct nvme_tcp_queue *queue)
+{
+	return queue->data_digest &&
+		test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags)
+		&& queue->sock && queue->sock->sk
+		&& ulp_ddp_need_map(queue->sock->sk);
+
+}
+#else
+
+static int nvme_tcp_map_pdu_info(struct nvme_tcp_queue *queue,
+				  size_t sent_size,
+				  u32 pdu_len, u32 data_len)
+{
+	return 0;
+}
+
+static void nvme_tcp_close_pdu_info(struct nvme_tcp_queue *queue)
+{
+}
+
+bool nvme_tcp_need_map(struct nvme_tcp_queue *queue)
+{
+	return false;
+}
+#endif
+
+
+
 static inline struct nvme_tcp_ctrl *to_tcp_ctrl(struct nvme_ctrl *ctrl)
 {
 	return container_of(ctrl, struct nvme_tcp_ctrl, ctrl);
@@ -285,11 +334,13 @@ static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash,
 
 #ifdef CONFIG_ULP_DDP
 
+void nvme_tcp_ddp_ddgst_fallback(struct ulp_ddp_pdu_info *pdu_info);
 static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
 static void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
 static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
 	.resync_request		= nvme_tcp_resync_request,
 	.ddp_teardown_done	= nvme_tcp_ddp_teardown_done,
+	.ddp_ddgst_fallback	= nvme_tcp_ddp_ddgst_fallback,
 };
 
 static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
@@ -371,6 +422,12 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
 
 	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
 	if (netdev->features & NETIF_F_HW_ULP_DDP) {
+		if (ulp_ddp_init_tx_offload(queue->sock->sk)) {
+			netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk);
+			dev_put(netdev);
+			return -ENOMEM;
+		}
+
 		set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
 		set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
 		set_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags);
@@ -392,6 +449,9 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
 	clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
 	clear_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags);
 
+	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags))
+		ulp_ddp_release_tx_offload(queue->sock->sk);
+
 	netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk);
 
 	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL;
@@ -1269,6 +1329,19 @@ static void nvme_tcp_fail_request(struct nvme_tcp_request *req)
 	nvme_tcp_end_request(blk_mq_rq_from_pdu(req), NVME_SC_HOST_PATH_ERROR);
 }
 
+#ifdef CONFIG_ULP_DDP
+void nvme_tcp_ddp_ddgst_fallback(struct ulp_ddp_pdu_info *pdu_info)
+{
+	struct request *rq = pdu_info->req;
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_queue *queue = req->queue;
+
+	nvme_tcp_ddp_ddgst_recalc(queue->snd_hash, rq);
+	nvme_tcp_ddgst_final(queue->snd_hash, &pdu_info->ddgst);
+}
+
+#endif
+
 static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 {
 	struct nvme_tcp_queue *queue = req->queue;
@@ -1333,7 +1406,8 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	int len = sizeof(*pdu) + hdgst - req->offset;
 	struct request *rq = blk_mq_rq_from_pdu(req);
 	int flags = MSG_DONTWAIT;
-	int ret;
+	int ret, check;
+	u32 data_len;
 
 	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest)
 		flags |= MSG_DDP_CRC;
@@ -1353,6 +1427,13 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	if (unlikely(ret <= 0))
 		return ret;
 
+	if (nvme_tcp_need_map(queue)) {
+		data_len = inline_data ? req->data_len : 0;
+		check = nvme_tcp_map_pdu_info(queue, ret, len, data_len);
+		if (unlikely(check))
+			return check;
+	}
+
 	len -= ret;
 	if (!len) {
 		if (inline_data) {
@@ -1360,6 +1441,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 			if (queue->data_digest)
 				crypto_ahash_init(queue->snd_hash);
 		} else {
+			nvme_tcp_close_pdu_info(queue);
 			nvme_tcp_done_send_req(queue);
 		}
 		return 1;
@@ -1376,7 +1458,7 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)
 	struct nvme_tcp_data_pdu *pdu = req->pdu;
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) - req->offset + hdgst;
-	int ret;
+	int ret, check;
 
 	if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest)
 		flags |= MSG_DDP_CRC;
@@ -1389,6 +1471,12 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)
 	if (unlikely(ret <= 0))
 		return ret;
 
+	if (nvme_tcp_need_map(queue)) {
+		check = nvme_tcp_map_pdu_info(queue, ret, len, req->data_len);
+		if (unlikely(check))
+			return check;
+	}
+
 	len -= ret;
 	if (!len) {
 		req->state = NVME_TCP_SEND_DATA;
@@ -1424,6 +1512,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 		return ret;
 
 	if (req->offset + ret == NVME_TCP_DIGEST_LENGTH) {
+		nvme_tcp_close_pdu_info(queue);
 		nvme_tcp_done_send_req(queue);
 		return 1;
 	}
Signature · 1 line
2.24.1
author reviewer
#48 lore raw

PATCH v5 27/36 mlx5e: make preparation in TLS code for NVMEoTCP CRC Tx offload

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

NVMEoTCP CRC Tx offload is similar to TLS Tx offload, and uses DUMP wqe as well.

To avoid duplicate functions the following changes were added:

1. Add DUMP_WQE.type field (=TLS or NVMEoTCP).
2. change in mlx5e_ktls_tx_handle_resync_dump_comp to handle also NVMEoTCP Tx DUMP WQE.

---
 drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h    |  5 +++++
 .../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c   | 12 ++++++++++--
 2 files changed, 15 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +5 -0
index f0190ee6e42c..c7f979dfdd69 100644
@@ -77,6 +77,10 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
 netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev);
 bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget);
 void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq);
+enum mlx5e_dump_wqe_type {
+	MLX5E_DUMP_WQE_TLS,
+	MLX5E_DUMP_WQE_NVMEOTCP,
+};
 
 static inline bool
 mlx5e_wqc_has_room_for(struct mlx5_wq_cyc *wq, u16 cc, u16 pc, u16 n)
@@ -140,6 +144,7 @@ struct mlx5e_tx_wqe_info {
 	u8 num_fifo_pkts;
 #ifdef CONFIG_MLX5_EN_TLS
 	struct page *resync_dump_frag_page;
+	enum mlx5e_dump_wqe_type type;
 #endif
 };
 
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +10 -2
index 9ad3459fb63a..64780d0143ec 100644
@@ -154,6 +154,7 @@ static void tx_fill_wi(struct mlx5e_txqsq *sq,
 		.num_wqebbs = num_wqebbs,
 		.num_bytes  = num_bytes,
 		.resync_dump_frag_page = page,
+		.type = MLX5E_DUMP_WQE_TLS,
 	};
 }
 
@@ -358,8 +359,15 @@ void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
 
 	mlx5e_tx_dma_unmap(sq->pdev, dma);
 	put_page(wi->resync_dump_frag_page);
-	stats->tls_dump_packets++;
-	stats->tls_dump_bytes += wi->num_bytes;
+
+	switch (wi->type) {
+	case MLX5E_DUMP_WQE_TLS:
+		stats->tls_dump_packets++;
+		stats->tls_dump_bytes += wi->num_bytes;
+		break;
+	case MLX5E_DUMP_WQE_NVMEOTCP:
+		break;
+	}
 }
 
 static void tx_post_fence_nop(struct mlx5e_txqsq *sq)
Signature · 1 line
2.24.1
author reviewer
#49 lore raw

PATCH v5 28/36 mlx5: Add sq state test bit for nvmeotcp

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

Add to the sq state a bit indicate if their is an nvmeotcp crc tx offload.

---
 drivers/net/ethernet/mellanox/mlx5/core/en.h      | 1 +
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++
 2 files changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
index c87f32492ea4..58724d4c27e1 100644
@@ -353,6 +353,7 @@ enum {
 	MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE,
 	MLX5E_SQ_STATE_PENDING_XSK_TX,
 	MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC,
+	MLX5E_SQ_STATE_NVMEOTCP,
 };
 
 struct mlx5e_tx_mpwqe {
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -0
index ecb12c7fdb7d..b76e590c237b 100644
@@ -1142,6 +1142,8 @@ static int mlx5e_alloc_txqsq(struct mlx5e_channel *c,
 		set_bit(MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE, &sq->state);
 	if (MLX5_IPSEC_DEV(c->priv->mdev))
 		set_bit(MLX5E_SQ_STATE_IPSEC, &sq->state);
+	if (MLX5_CAP_DEV_NVMEOTCP(c->priv->mdev, crc_tx))
+		set_bit(MLX5E_SQ_STATE_NVMEOTCP, &sq->state);
 	if (param->is_mpw)
 		set_bit(MLX5E_SQ_STATE_MPWQE, &sq->state);
 	sq->stop_room = param->stop_room;
Signature · 1 line
2.24.1
author reviewer
#50 lore raw

PATCH v5 29/36 mlx5: Add support to NETIF_F_HW_TCP_DDP_CRC_TX feature

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

---
 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +5 -0
index 7f6607cac9fb..db6ca734d129 100644
@@ -961,6 +961,11 @@ void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv)
 		netdev->hw_features |= NETIF_F_HW_ULP_DDP;
 	}
 
+	if (MLX5_CAP_DEV_NVMEOTCP(priv->mdev, crc_tx)) {
+		netdev->features |= NETIF_F_HW_ULP_DDP;
+		netdev->hw_features |= NETIF_F_HW_ULP_DDP;
+	}
+
 	netdev->ulp_ddp_ops = &mlx5e_nvmeotcp_ops;
 }
 
Signature · 1 line
2.24.1
author reviewer
#51 lore raw

PATCH v5 30/36 net/mlx5e: NVMEoTCP DDGST TX offload TIS

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

NVMEoTCP DDGST Tx offload needs TIS.
This commit add the infrastructer for this TIS.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c      | 17 +++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/en_main.c   |  2 +-
 include/linux/mlx5/mlx5_ifc.h                   |  3 ++-
 3 files changed, 20 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +17 -0
index db6ca734d129..d42f346ac8f5 100644
@@ -35,6 +35,11 @@ static void mlx5e_nvmeotcp_destroy_tir(struct mlx5e_priv *priv, int tirn)
 	mlx5_core_destroy_tir(priv->mdev, tirn);
 }
 
+static void mlx5e_nvmeotcp_delete_tis(struct mlx5e_priv *priv, int tisn)
+{
+	mlx5_core_destroy_tis(priv->mdev, tisn);
+}
+
 static inline u32
 mlx5e_get_channel_ix_from_io_cpu(struct mlx5e_priv *priv, u32 io_cpu)
 {
@@ -137,6 +142,18 @@ void mlx5_destroy_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev, u32 uid)
 	mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
 }
 
+static int mlx5e_nvmeotcp_create_tis(struct mlx5_core_dev *mdev, u32 *tisn)
+{
+	u32 in[MLX5_ST_SZ_DW(create_tis_in)] = {};
+	void *tisc;
+
+	tisc = MLX5_ADDR_OF(create_tis_in, in, ctx);
+
+	MLX5_SET(tisc, tisc, nvmeotcp_en, 1);
+
+	return mlx5e_create_tis(mdev, in, tisn);
+}
+
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS 0x2
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -1
index b76e590c237b..2a9718d3c2d3 100644
@@ -3152,7 +3152,7 @@ int mlx5e_create_tis(struct mlx5_core_dev *mdev, void *in, u32 *tisn)
 
 	MLX5_SET(tisc, tisc, transport_domain, mdev->mlx5e_res.hw_objs.td.tdn);
 
-	if (MLX5_GET(tisc, tisc, tls_en))
+	if (MLX5_GET(tisc, tisc, tls_en) || MLX5_GET(tisc, tisc, nvmeotcp_en))
 		MLX5_SET(tisc, tisc, pd, mdev->mlx5e_res.hw_objs.pdn);
 
 	if (mlx5_lag_is_lacp_owner(mdev))
include/linux/mlx5/mlx5_ifc.h +2 -1
index f0310c24f408..a4965bf1e607 100644
@@ -3300,7 +3300,8 @@ struct mlx5_ifc_traffic_counter_bits {
 struct mlx5_ifc_tisc_bits {
 	u8         strict_lag_tx_port_affinity[0x1];
 	u8         tls_en[0x1];
-	u8         reserved_at_2[0x2];
+	u8         nvmeotcp_en[0x1];
+	u8         reserved_at_3[0x1];
 	u8         lag_tx_port_affinity[0x04];
 
 	u8         reserved_at_8[0x4];
Signature · 1 line
2.24.1
author reviewer
#52 lore raw

PATCH v5 31/36 net/mlx5e: NVMEoTCP DDGST Tx offload queue init/teardown

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

This commit add support for DDGST TX offload to the mlx5e_nvmeotcp_queue_init/teardown function. If enable, mlx5e_nvmeotcp_queue_init will call mlx5e_nvmeotcp_queue_tx_init to handle TX offload init.

Add to mlx5e NVMEoTCP queue is responsible for:
- Create a separate TIS to identify the queue and maintain the HW context
- Update ulp_ddp_ctx params.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 47 +++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    | 12 ++++-
 2 files changed, 58 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +47 -0
index d42f346ac8f5..6023e1ae7be4 100644
@@ -669,6 +669,36 @@ mlx5e_nvmeotcp_queue_rx_init(struct mlx5e_nvmeotcp_queue *queue,
 	return err;
 }
 
+static int
+mlx5e_nvmeotcp_queue_tx_init(struct mlx5e_nvmeotcp_queue *queue,
+			     struct mlx5_core_dev *mdev,
+			     struct net_device *netdev)
+{
+	struct sock *sk = queue->sk;
+	int err, tisn;
+
+	err = mlx5e_nvmeotcp_create_tis(mdev, &tisn);
+
+	if (err) {
+		mlx5_core_err(mdev, "create tis failed, %d\n", err);
+		return err;
+	}
+
+	queue->tisn = tisn;
+	queue->ulp_ddp_ctx.expected_seq = tcp_sk(sk)->write_seq;
+	queue->pending = true;
+	queue->end_seq_hint = 0;
+	queue->ulp_ddp_ctx.netdev = netdev;
+	queue->ulp_ddp_ctx.ddgst_len = 4;
+
+	/* following this assignment mlx5e_nvmeotcp_is_sk_tx_device_offloaded
+	 * will return true and ulp_ddp_ctx might be accessed
+	 * by the netdev's xmit function.
+	 */
+	smp_store_release(&sk->sk_validate_xmit_skb, ulp_ddp_validate_xmit_skb);
+	return err;
+}
+
 #define OCTWORD_SHIFT 4
 #define MAX_DS_VALUE 63
 static int
@@ -680,6 +710,8 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	bool crc_rx = ((netdev->features & NETIF_F_HW_ULP_DDP) &&
 		       (config->dgst & NVME_TCP_DATA_DIGEST_ENABLE));
 	bool zerocopy = (netdev->features & NETIF_F_HW_ULP_DDP);
+	bool crc_tx = (config->dgst & NVME_TCP_DATA_DIGEST_ENABLE)  &&
+		(netdev->features & NETIF_F_HW_ULP_DDP);
 	struct mlx5e_priv *priv = netdev_priv(netdev);
 	struct mlx5_core_dev *mdev = priv->mdev;
 	struct mlx5e_nvmeotcp_queue *queue;
@@ -709,6 +741,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 		goto free_queue;
 	}
 
+	queue->crc_tx = crc_tx;
 	queue->crc_rx = crc_rx;
 	queue->zerocopy = zerocopy;
 	queue->ulp_ddp_ctx.type = ULP_DDP_NVME;
@@ -736,6 +769,12 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	if (err)
 		goto destroy_rx;
 
+	if (crc_tx) {
+		err = mlx5e_nvmeotcp_queue_tx_init(queue, mdev, netdev);
+		if (err)
+			goto remove_queue_from_hash;
+	}
+
 	stats->nvmeotcp_queue_init++;
 	write_lock_bh(&sk->sk_callback_lock);
 	ulp_ddp_set_ctx(sk, queue);
@@ -743,6 +782,9 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
 	refcount_set(&queue->ref_count, 1);
 	return err;
 
+remove_queue_from_hash:
+	rhashtable_remove_fast(&priv->nvmeotcp->queue_hash,
+			       &queue->hash, rhash_queues);
 destroy_rx:
 	if (zerocopy || crc_rx)
 		mlx5e_nvmeotcp_destroy_rx(queue, mdev, zerocopy);
@@ -778,6 +820,11 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
 	rhashtable_remove_fast(&priv->nvmeotcp->queue_hash, &queue->hash,
 			       rhash_queues);
 	ida_simple_remove(&priv->nvmeotcp->queue_ids, queue->id);
+	if (queue->crc_tx) {
+		smp_store_release(&sk->sk_validate_xmit_skb, NULL);
+		mlx5e_nvmeotcp_delete_tis(priv, queue->tisn);
+	}
+
 	write_lock_bh(&sk->sk_callback_lock);
 	ulp_ddp_set_ctx(sk, NULL);
 	write_unlock_bh(&sk->sk_callback_lock);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +11 -1
index b9642e130b97..3bc45b81da06 100644
@@ -47,12 +47,16 @@ struct mlx5e_nvmeotcp_sq {
  *	@sk: The socket used by the NVMe-TCP queue
  *	@zerocopy: if this queue is used for zerocopy offload.
  *	@crc_rx: if this queue is used for CRC Rx offload.
+ *	@crc_tx: if this queue is used for CRC Tx offload.
  *	@ccid: ID of the current CC
  *	@ccsglidx: Index within the scatter-gather list (SGL) of the current CC
  *	@ccoff_inner: Current offset within the @ccsglidx element
  *	@priv: mlx5e netdev priv
  *	@inv_done: invalidate callback of the nvme tcp driver
  *	@after_resync_cqe: indicate if resync occurred
+ *	@tisn: Destination TIS number created for NVMEoTCP CRC TX offload
+ *	@pending: indicate if static/progress params need to be send to NIC.
+ *	@end_seq_hint: Tx ooo - offload packet only if it ends after the hint.
  */
 struct mlx5e_nvmeotcp_queue {
 	struct ulp_ddp_ctx		ulp_ddp_ctx;
@@ -66,7 +70,7 @@ struct mlx5e_nvmeotcp_queue {
 	u32				tag_buf_table_id;
 	struct rhash_head		hash;
 	refcount_t			ref_count;
-	bool				dgst;
+	int				dgst;
 	int				pda;
 	u32				ccid_gen;
 	u32				max_klms_per_wqe;
@@ -74,6 +78,12 @@ struct mlx5e_nvmeotcp_queue {
 	struct sock			*sk;
 	bool				zerocopy;
 	bool				crc_rx;
+	bool				crc_tx;
+	/* for crc_tx offload */
+	int				tisn;
+	bool				pending;
+	u32				end_seq_hint;
+	u32				start_pdu_hint;
 
 	/* current ccid fields */
 	off_t				ccoff;
Signature · 1 line
2.24.1
author reviewer
#53 lore raw

PATCH v5 32/36 net/mlx5e: NVMEoTCP DDGST TX BSF and PSV

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

Change the function that build NVMEoTCP progress params and static params, to work for Tx/Rx.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 130 ++++++++++++++----
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |   4 +-
 2 files changed, 108 insertions(+), 26 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +106 -24
index 6023e1ae7be4..624d8a28dc21 100644
@@ -155,8 +155,11 @@ static int mlx5e_nvmeotcp_create_tis(struct mlx5_core_dev *mdev, u32 *tisn)
 }
 
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIS_PARAMS 0x1
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS 0x2
 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_STATIC_PARAMS 0x1
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_PROGRESS_PARAMS 0x3
 
 #define STATIC_PARAMS_DS_CNT \
 	DIV_ROUND_UP(MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS)
@@ -250,56 +253,75 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue,
 static void
 fill_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
 			      struct mlx5_seg_nvmeotcp_progress_params *params,
-			      u32 seq)
+			      u32 seq, bool is_tx)
 {
 	void *ctx = params->ctx;
 
-	params->tir_num = cpu_to_be32(queue->tirn);
+	params->tir_num = is_tx ? cpu_to_be32(queue->tisn) : cpu_to_be32(queue->tirn);
 
 	MLX5_SET(nvmeotcp_progress_params, ctx,
 		 next_pdu_tcp_sn, seq);
 	MLX5_SET(nvmeotcp_progress_params, ctx, pdu_tracker_state,
 		 MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START);
+	if (is_tx)
+		MLX5_SET(nvmeotcp_progress_params, ctx, offloading_state, 0);
+}
+
+static void nvme_tx_fill_wi(struct mlx5e_txqsq *sq,
+			    u16 pi, u8 num_wqebbs, u32 num_bytes,
+			    struct page *page, enum mlx5e_dump_wqe_type type)
+{
+	struct mlx5e_tx_wqe_info *wi = &sq->db.wqe_info[pi];
+
+	*wi = (struct mlx5e_tx_wqe_info) {
+		.num_wqebbs = num_wqebbs,
+		.num_bytes  = num_bytes,
+	};
 }
 
 void
 build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
 			       struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe,
-			       u32 seq)
+			       u32 seq, bool is_rx, bool resync, u16 pc, u32 sqn)
 {
 	struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
-	u32 sqn = queue->sq->icosq.sqn;
-	u16 pc = queue->sq->icosq.pc;
-	u8 opc_mod;
+	u8 opc_mod = is_rx ?
+		MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS :
+		MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_PROGRESS_PARAMS;
 
 	memset(wqe, 0, MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ);
-	opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS;
+
 	cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
 					     MLX5_OPCODE_SET_PSV | (opc_mod << 24));
 	cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
 				   PROGRESS_PARAMS_DS_CNT);
-	fill_nvmeotcp_progress_params(queue, &wqe->params, seq);
+	fill_nvmeotcp_progress_params(queue, &wqe->params, seq, !is_rx);
 }
 
 static void
 fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
 			    struct mlx5_seg_nvmeotcp_static_params *params,
-			    u32 resync_seq, bool zero_copy_en,
+			    u32 resync_seq, bool is_rx, bool zero_copy_en,
 			    bool ddgst_offload_en)
 {
 	void *ctx = params->ctx;
+	int pda = queue->pda;
+	bool hddgst_en = queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE;
+	bool ddgst_en = queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE;
+
+	if (!is_rx) {
+		pda = 0;
+	}
 
 	MLX5_SET(transport_static_params, ctx, const_1, 1);
 	MLX5_SET(transport_static_params, ctx, const_2, 2);
 	MLX5_SET(transport_static_params, ctx, acc_type,
 		 MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP);
 	MLX5_SET(transport_static_params, ctx, nvme_resync_tcp_sn, resync_seq);
-	MLX5_SET(transport_static_params, ctx, pda, queue->pda);
-	MLX5_SET(transport_static_params, ctx, ddgst_en,
-		 queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE);
+	MLX5_SET(transport_static_params, ctx, pda, pda);
+	MLX5_SET(transport_static_params, ctx, ddgst_en, ddgst_en);
 	MLX5_SET(transport_static_params, ctx, ddgst_offload_en, ddgst_offload_en);
-	MLX5_SET(transport_static_params, ctx, hddgst_en,
-		 queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE);
+	MLX5_SET(transport_static_params, ctx, hddgst_en, hddgst_en);
 	MLX5_SET(transport_static_params, ctx, hdgst_offload_en, 0);
 	MLX5_SET(transport_static_params, ctx, ti,
 		 MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR);
@@ -310,26 +332,31 @@ fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
 void
 build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
 			     struct mlx5e_set_nvmeotcp_static_params_wqe *wqe,
-			     u32 resync_seq, bool zerocopy, bool crc_rx)
+			     u32 resync_seq, bool is_rx, u16 pc, u32 sqn,
+			     bool zerocopy, bool crc_rx)
 {
-	u8 opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS;
 	struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
 	struct mlx5_wqe_ctrl_seg      *cseg = &wqe->ctrl;
-	u32 sqn = queue->sq->icosq.sqn;
-	u16 pc = queue->sq->icosq.pc;
+	int tirn_tisn = is_rx ? queue->tirn : queue->tisn;
+	u8 opc_mod = is_rx ?
+		MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS :
+		MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_STATIC_PARAMS;
+
 
 	memset(wqe, 0, MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ);
 
-	cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
-					     MLX5_OPCODE_UMR | (opc_mod) << 24);
+	cseg->opmod_idx_opcode  = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
+					      MLX5_OPCODE_UMR | (opc_mod) << 24);
 	cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
 				   STATIC_PARAMS_DS_CNT);
-	cseg->imm = cpu_to_be32(queue->tirn << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT);
+	cseg->imm = cpu_to_be32(tirn_tisn <<
+				MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT);
 
 	ucseg->flags = MLX5_UMR_INLINE;
 	ucseg->bsf_octowords =
 		cpu_to_be16(MLX5E_NVMEOTCP_STATIC_PARAMS_OCTWORD_SIZE);
-	fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq, zerocopy, crc_rx);
+	fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq,
+				    is_rx, zerocopy, crc_rx);
 }
 
 static void
@@ -371,7 +398,8 @@ mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue,
 	pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs);
 	wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi);
 	mlx5e_nvmeotcp_fill_wi(NULL, sq, wqe_bbs, pi, 0, BSF_UMR);
-	build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->zerocopy, queue->crc_rx);
+	build_nvmeotcp_static_params(queue, wqe, resync_seq, true, sq->pc,
+				     sq->sqn, queue->zerocopy, queue->crc_rx);
 	sq->pc += wqe_bbs;
 	mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
 	spin_unlock(&queue->nvmeotcp_icosq_lock);
@@ -389,7 +417,7 @@ mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue,
 	pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs);
 	wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi);
 	mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, 0, SET_PSV_UMR);
-	build_nvmeotcp_progress_params(queue, wqe, seq);
+	build_nvmeotcp_progress_params(queue, wqe, seq, true, false, sq->pc, sq->sqn);
 	sq->pc += wqe_bbs;
 	mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
 }
@@ -1078,6 +1106,60 @@ int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv)
 	return ret;
 }
 
+static
+void mlx5e_nvmeotcp_tx_post_static_params(struct mlx5e_nvmeotcp_queue *queue,
+					  struct mlx5e_txqsq *sq)
+{
+	struct mlx5e_set_nvmeotcp_static_params_wqe *wqe;
+	enum mlx5e_dump_wqe_type type = MLX5E_DUMP_WQE_NVMEOTCP;
+	u16 pi, wqe_bbs;
+
+	wqe_bbs = MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS;
+	pi = mlx5e_txqsq_get_next_pi(sq, wqe_bbs);
+	wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi);
+	nvme_tx_fill_wi(sq, pi, wqe_bbs, 0, NULL, type);
+	build_nvmeotcp_static_params(queue, wqe, 0, false,
+				     sq->pc, sq->sqn, false, true);
+	sq->pc += wqe_bbs;
+	mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
+}
+
+static
+void mlx5e_nvmeotcp_tx_post_progress_params(struct mlx5e_nvmeotcp_queue *queue,
+					    struct mlx5e_txqsq *sq, u32 seq,
+					    bool resync)
+{
+	struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe;
+	enum mlx5e_dump_wqe_type type = MLX5E_DUMP_WQE_NVMEOTCP;
+	u16 pi, wqe_bbs;
+
+	wqe_bbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS;
+	pi = mlx5e_txqsq_get_next_pi(sq, wqe_bbs);
+	wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi);
+	nvme_tx_fill_wi(sq, pi, wqe_bbs, 0, NULL, type);
+	build_nvmeotcp_progress_params(queue, wqe, seq, false, resync, sq->pc, sq->sqn);
+	sq->pc += wqe_bbs;
+	mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
+}
+
+static
+bool mlx5e_nvmeotcp_test_and_clear_pending(struct mlx5e_nvmeotcp_queue *ctx)
+{
+	bool ret = ctx->pending;
+
+	ctx->pending = false;
+
+	return ret;
+}
+
+static
+void mlx5e_nvmeotcp_tx_post_param_wqes(struct mlx5e_txqsq *sq, struct sock *sk,
+				       struct mlx5e_nvmeotcp_queue *ctx)
+{
+	mlx5e_nvmeotcp_tx_post_static_params(ctx, sq);
+	mlx5e_nvmeotcp_tx_post_progress_params(ctx, sq, tcp_sk(sk)->copied_seq, false);
+}
+
 void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv)
 {
 	struct mlx5e_nvmeotcp *nvmeotcp = priv->nvmeotcp;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h +2 -2
index 44671e28a9ea..e7436aa01ad4 100644
@@ -69,12 +69,12 @@ struct mlx5e_get_psv_wqe {
 void
 build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
 			       struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe,
-			       u32 seq);
+			       u32 seq, bool is_rx, bool is_resync, u16 pc, u32 sqn);
 
 void
 build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
 			     struct mlx5e_set_nvmeotcp_static_params_wqe *wqe,
-			     u32 resync_seq,
+			     u32 resync_seq, bool is_rx, u16 pc, u32 sqn,
 			     bool zerocopy, bool crc_rx);
 
 #endif /* __MLX5E_NVMEOTCP_UTILS_H__ */
Signature · 1 line
2.24.1
author reviewer
#54 lore raw

PATCH v5 33/36 net/mlx5e: NVMEoTCP DDGST TX Data path

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

This patch handles only the good flow for the DDGST Tx offload
skb and wqe.
Later patches will handle bad flow (OOO packets)

1. add mlx5e_nvmeotcp_handle_tx_skb function(skb,...):
check if the skb can be offlaoded. this function track the tcp_seq of the skb, and check if this is the next tcp_seq. and if so, send this skb with DDGST Tx offload.

2. add mlx5e_nvmeotcp_handle_tx_wqe function :
send the wqe with the correct tis number for the offload.

---
 .../mellanox/mlx5/core/en_accel/en_accel.h    | 13 ++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 63 +++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    |  6 ++
 .../net/ethernet/mellanox/mlx5/core/en_tx.c   | 11 ++++
 4 files changed, 93 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +13 -0
index b9404366e6e8..2e5a7741736f 100644
@@ -113,6 +113,9 @@ struct mlx5e_accel_tx_state {
 #ifdef CONFIG_MLX5_EN_IPSEC
 	struct mlx5e_accel_tx_ipsec_state ipsec;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	int tisn;
+#endif
 };
 
 static inline bool mlx5e_accel_tx_begin(struct net_device *dev,
@@ -137,6 +140,12 @@ static inline bool mlx5e_accel_tx_begin(struct net_device *dev,
 	}
 #endif
 
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	if (test_bit(MLX5E_SQ_STATE_NVMEOTCP, &sq->state)) {
+		if (unlikely(!mlx5e_nvmeotcp_handle_tx_skb(dev, sq, skb, &state->tisn)))
+			return false;
+	}
+#endif
 	return true;
 }
 
@@ -187,6 +196,10 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,
 	mlx5e_tls_handle_tx_wqe(&wqe->ctrl, &state->tls);
 #endif
 
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	mlx5e_nvmeotcp_handle_tx_wqe(sq, &wqe->ctrl, state->tisn);
+#endif
+
 #ifdef CONFIG_MLX5_EN_IPSEC
 	if (test_bit(MLX5E_SQ_STATE_IPSEC, &sq->state) &&
 	    state->ipsec.xo && state->ipsec.tailen)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +63 -0

69 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +6 -0
index 3bc45b81da06..0451b3ac3687 100644
@@ -119,6 +119,12 @@ void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi);
 void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi);
 int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv);
 void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
+bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev,
+				  struct mlx5e_txqsq *sq,
+				  struct sk_buff *skb, int *tisn);
+void mlx5e_nvmeotcp_handle_tx_wqe(struct mlx5e_txqsq *sq,
+				  struct mlx5_wqe_ctrl_seg *csegl,
+				  int tisn);
 #else
 
 static inline void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv) { }
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +11 -0
index c63d78eda606..7feaf5608b9d 100644
@@ -249,6 +249,13 @@ mlx5e_txwqe_build_eseg_csum(struct mlx5e_txqsq *sq, struct sk_buff *skb,
 		eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;
 		sq->stats->csum_partial++;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	} else if (unlikely(accel && accel->tisn)) {
+		eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;
+		sq->stats->csum_partial++;
+#endif
+
+
 	} else if (unlikely(mlx5e_ipsec_eseg_meta(eseg))) {
 		ipsec_txwqe_build_eseg_csum(sq, skb, eseg);
 	} else
@@ -352,6 +359,10 @@ mlx5e_tx_wqe_inline_mode(struct mlx5e_txqsq *sq, struct sk_buff *skb,
 	if (accel && accel->tls.tls_tisn)
 		return MLX5_INLINE_MODE_TCP_UDP;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	if (accel && accel->tisn)
+		return MLX5_INLINE_MODE_TCP_UDP;
+#endif
 
 	mode = sq->min_inline_mode;
 
Signature · 1 line
2.24.1
author reviewer
#55 lore raw

PATCH v5 34/36 net/mlx5e: NVMEoTCP DDGST TX handle OOO packets

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

When the driver indicate an OOO NVMEoTCP Tx packet it starts OOO flow:

1. Get pdu_info from nvme-tcp.
2. Send indication to NIC (set psv)- NIC will rebuild the parse machine.
3. Send the data the NIC needs for computing the DDGST using DUMP wqes.

---
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   2 +-
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 281 +++++++++++++++++-
 2 files changed, 280 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +1 -1
index c7f979dfdd69..1f4beaac488a 100644
@@ -142,7 +142,7 @@ struct mlx5e_tx_wqe_info {
 	u8 num_wqebbs;
 	u8 num_dma;
 	u8 num_fifo_pkts;
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined CONFIG_MLX5_EN_TLS || defined CONFIG_MLX5_EN_NVMEOTCP
 	struct page *resync_dump_frag_page;
 	enum mlx5e_dump_wqe_type type;
 #endif
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +279 -2

320 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#56 lore raw

PATCH v5 35/36 net/mlx5e: NVMEoTCP DDGST TX offload optimization

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

Stop offloading NVMEoTCP OOO packets which aren't contains the pdu DDGST field.

When the driver indicate an OOO NVMEoTCP packet he check if it contains NVMEoTCP PDU DDGST field, if so he offloads it, otherwise he isn't.

---
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +23 -0
index f8cba90679ea..f3ef92167e25 100644
@@ -1393,6 +1393,20 @@ bool mlx5e_nvmeotcp_resync_cap(struct mlx5e_nvmeotcp_queue *queue,
 	return false;
 }
 
+static
+bool mlx5e_nvmeotcp_check_if_need_offload(struct mlx5e_nvmeotcp_queue *queue,
+					  u32  end_seq, u32 start_seq)
+{
+	if (!queue->end_seq_hint)
+		return false;
+
+	/* check if skb end after pdu crc start */
+	if (before(queue->end_seq_hint, end_seq) || before(start_seq, queue->start_pdu_hint))
+		return false;
+
+	return true;
+}
+
 static enum mlx5e_nvmeotcp_resync_retval
 mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue,
 			      struct mlx5e_txqsq *sq, struct sk_buff *skb,
@@ -1400,12 +1414,21 @@ mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue,
 {
 	struct ulp_ddp_pdu_info *pdu_info = NULL;
 
+	if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) {
+		return MLX5E_NVMEOTCP_RESYNC_SKIP;
+
 	/* ask for pdu_info that includes the tcp_seq */
 	pdu_info = ulp_ddp_get_pdu_info(skb->sk, seq);
 
 	if (!pdu_info)
 		return MLX5E_NVMEOTCP_RESYNC_SKIP;
 
+	queue->end_seq_hint = pdu_info->end_seq - 4;
+	queue->start_pdu_hint = pdu_info->start_seq;
+	/* check if this packet contain crc - if so offload else no */
+	if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) {
+		return MLX5E_NVMEOTCP_RESYNC_SKIP;
+
 	/*update NIC about resync - he will rebuild parse machine
 	 *send psv with small fence
 	 */
Signature · 1 line
2.24.1
author reviewer
#57 lore raw

PATCH v5 36/36 net/mlx5e: NVMEoTCP DDGST TX statistics

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

NVMEoTCP Tx offload statistics includes both control and data path
statistic: counters for contex, offloaded packets/bytes, out-of-order packets resync operation (success/fail), and DUMP packets/bytes.

---
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     |  4 ++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 22 ++++++++++-
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 37 +++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/en_stats.h    | 23 ++++++++++++
 4 files changed, 84 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +4 -0
index 64780d0143ec..4b0d4bd88b9c 100644
@@ -366,6 +366,10 @@ void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
 		stats->tls_dump_bytes += wi->num_bytes;
 		break;
 	case MLX5E_DUMP_WQE_NVMEOTCP:
+#ifdef CONFIG_ULP_DDP
+		stats->nvmeotcp_dump_packets++;
+		stats->nvmeotcp_dump_bytes += wi->num_bytes;
+#endif
 		break;
 	}
 }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +20 -2
index f3ef92167e25..34676c81d889 100644
@@ -1388,8 +1388,10 @@ bool mlx5e_nvmeotcp_resync_cap(struct mlx5e_nvmeotcp_queue *queue,
 	if (unlikely(ret))
 		goto err_out;
 out:
+	sq->stats->nvmeotcp_resync++;
 	return true;
 err_out:
+	sq->stats->nvmeotcp_resync_fail++;
 	return false;
 }
 
@@ -1413,21 +1415,29 @@ mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue,
 			      u32  seq, int datalen)
 {
 	struct ulp_ddp_pdu_info *pdu_info = NULL;
+	struct mlx5e_sq_stats *stats = sq->stats;
 
+	stats->nvmeotcp_ooo++;
 	if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) {
+		stats->nvmeotcp_no_need_offload++;
 		return MLX5E_NVMEOTCP_RESYNC_SKIP;
+	}
 
 	/* ask for pdu_info that includes the tcp_seq */
 	pdu_info = ulp_ddp_get_pdu_info(skb->sk, seq);
 
-	if (!pdu_info)
+	if (!pdu_info) {
+		stats->nvmeotcp_no_pdu_info++;
 		return MLX5E_NVMEOTCP_RESYNC_SKIP;
+	}
 
 	queue->end_seq_hint = pdu_info->end_seq - 4;
 	queue->start_pdu_hint = pdu_info->start_seq;
 	/* check if this packet contain crc - if so offload else no */
 	if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) {
+		stats->nvmeotcp_no_need_offload++;
 		return MLX5E_NVMEOTCP_RESYNC_SKIP;
+	}
 
 	/*update NIC about resync - he will rebuild parse machine
 	 *send psv with small fence
@@ -1464,6 +1474,7 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev,
 				  struct sk_buff *skb, int *nvmeotcp_tisn)
 {
 	struct mlx5e_nvmeotcp_queue *ctx;
+	struct mlx5e_sq_stats *stats = sq->stats;
 	int datalen;
 	u32 seq;
 
@@ -1484,8 +1495,10 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev,
 	if (WARN_ON_ONCE(ctx->ulp_ddp_ctx.netdev != netdev))
 		goto err_out;
 
-	if (unlikely(mlx5e_nvmeotcp_test_and_clear_pending(ctx)))
+	if (unlikely(mlx5e_nvmeotcp_test_and_clear_pending(ctx))) {
 		mlx5e_nvmeotcp_tx_post_param_wqes(sq, skb->sk, ctx);
+		stats->nvmeotcp_ctx++;
+	}
 
 	seq = ntohl(tcp_hdr(skb)->seq);
 	if (unlikely(ctx->ulp_ddp_ctx.expected_seq != seq)) {
@@ -1504,6 +1517,11 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev,
 
 	*nvmeotcp_tisn = ctx->tisn;
 	ctx->ulp_ddp_ctx.expected_seq = seq + datalen;
+	stats->nvmeotcp_offload_packets += skb_is_gso(skb) ?
+		skb_shinfo(skb)->gso_segs : 1;
+
+	stats->nvmeotcp_offload_bytes   += datalen;
+
 	goto good_out;
 out:
 	*nvmeotcp_tisn = 0;
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +37 -0
index 3d95e46422e5..c51d1a76b22f 100644
@@ -125,6 +125,18 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_tls_drop_no_sync_data) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_tls_drop_bypass_req) },
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_offload_bytes) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_ooo) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_dump_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_dump_bytes) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_resync) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_ctx) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_resync_fail) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_no_need_offload) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_no_pdu_info) },
+#endif
 
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_lro_packets) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_lro_bytes) },
@@ -420,6 +432,19 @@ static void mlx5e_stats_grp_sw_update_stats_sq(struct mlx5e_sw_stats *s,
 	s->tx_tls_drop_no_sync_data += sq_stats->tls_drop_no_sync_data;
 	s->tx_tls_drop_bypass_req   += sq_stats->tls_drop_bypass_req;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	s->tx_nvmeotcp_offload_packets += sq_stats->nvmeotcp_offload_packets;
+	s->tx_nvmeotcp_offload_bytes   += sq_stats->nvmeotcp_offload_bytes;
+	s->tx_nvmeotcp_ooo             += sq_stats->nvmeotcp_ooo;
+	s->tx_nvmeotcp_dump_bytes      += sq_stats->nvmeotcp_dump_bytes;
+	s->tx_nvmeotcp_dump_packets    += sq_stats->nvmeotcp_dump_packets;
+	s->tx_nvmeotcp_resync	       += sq_stats->nvmeotcp_resync;
+	s->tx_nvmeotcp_ctx             += sq_stats->nvmeotcp_ctx;
+	s->tx_nvmeotcp_resync_fail     += sq_stats->nvmeotcp_resync_fail;
+	s->tx_nvmeotcp_no_need_offload += sq_stats->nvmeotcp_no_need_offload;
+	s->tx_nvmeotcp_no_pdu_info     += sq_stats->nvmeotcp_no_pdu_info;
+#endif
+
 	s->tx_cqes                  += sq_stats->cqes;
 }
 
@@ -1850,6 +1875,18 @@ static const struct counter_desc sq_stats_desc[] = {
 	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_skip_no_sync_data) },
 	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_drop_no_sync_data) },
 	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_drop_bypass_req) },
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_offload_bytes) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_ooo) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_dump_packets) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_dump_bytes) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_resync) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_ctx) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_no_need_offload) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_no_pdu_info) },
+	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_resync_fail) },
 #endif
 	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, csum_none) },
 	{ MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, stopped) },
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +23 -0
index e645ee83de97..3ca48d69a2d6 100644
@@ -205,6 +205,17 @@ struct mlx5e_sw_stats {
 	u64 rx_nvmeotcp_resync;
 	u64 rx_nvmeotcp_offload_packets;
 	u64 rx_nvmeotcp_offload_bytes;
+
+	u64 tx_nvmeotcp_offload_packets;
+	u64 tx_nvmeotcp_offload_bytes;
+	u64 tx_nvmeotcp_ooo;
+	u64 tx_nvmeotcp_resync;
+	u64 tx_nvmeotcp_dump_packets;
+	u64 tx_nvmeotcp_dump_bytes;
+	u64 tx_nvmeotcp_ctx;
+	u64 tx_nvmeotcp_no_need_offload;
+	u64 tx_nvmeotcp_no_pdu_info;
+	u64 tx_nvmeotcp_resync_fail;
 #endif
 	u64 ch_events;
 	u64 ch_poll;
@@ -405,6 +416,18 @@ struct mlx5e_sq_stats {
 	u64 tls_skip_no_sync_data;
 	u64 tls_drop_no_sync_data;
 	u64 tls_drop_bypass_req;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	u64 nvmeotcp_offload_packets;
+	u64 nvmeotcp_offload_bytes;
+	u64 nvmeotcp_ooo;
+	u64 nvmeotcp_resync;
+	u64 nvmeotcp_dump_packets;
+	u64 nvmeotcp_dump_bytes;
+	u64 nvmeotcp_ctx;
+	u64 nvmeotcp_resync_fail;
+	u64 nvmeotcp_no_need_offload;
+	u64 nvmeotcp_no_pdu_info;
 #endif
 	/* less likely accessed in data path */
 	u64 csum_none;
Signature · 1 line
2.24.1
#2 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

In reply to Boris Pismenny
Quoted 4 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

...

Quoted 8 lines

};

const char

net/ipv4/tcp_input.c +0 -0
index e6ca5a1f3b59..4a7160bba09b 100644

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5154

Quoted 6 lines
net/ipv4/tcp_input.c +3 -0
@@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
                memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
                nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+               nskb->ddp_crc = skb->ddp_crc;

Probably you do not want to attempt any collapse if skb->ddp_crc is set right there.

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5191

Quoted 11 lines
net/ipv4/tcp_input.c +4 -0
 #endif
                TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
                if (list)
@@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
                                if (skb->decrypted != nskb->decrypted)
                                        goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+
+                               if (skb->ddp_crc != nskb->ddp_crc)

This checks only the second, third, and remaining skbs.

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5195

Quoted 4 lines
net/ipv4/tcp_input.c +1 -0
+                                       goto end;
 #endif
                        }
                }

tcp_collapse() is copying data from small skbs to pack it to bigger skb (one page of payload), in case
of memory emergency/pressure (socket queues are full)

If your changes are trying to avoid 'needless' copies, maybe you should reconsider and let the emergency packing be done.

If the copy is not _possible_, you should rephrase your changelog to clearly state the kernel _cannot_ access this memory in any way.

#3 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

Quoted 9 lines · Eric Dumazet
Quoted 4 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

...

Quoted 2 lines

};

Quoted 16 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5154

Quoted 12 lines

const char

net/ipv4/tcp_input.c +3 -0
index e6ca5a1f3b59..4a7160bba09b 100644
@@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
                memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
                nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+               nskb->ddp_crc = skb->ddp_crc;

Probably you do not want to attempt any collapse if skb->ddp_crc is set right there.

Right.

Quoted 5 lines
Quoted 5 lines
net/ipv4/tcp_input.c +0 -0
 #endif
                TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
                if (list)
@@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
Quoted 9 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5191

Quoted 6 lines
net/ipv4/tcp_input.c +4 -0
                                if (skb->decrypted != nskb->decrypted)
                                        goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+
+                               if (skb->ddp_crc != nskb->ddp_crc)

This checks only the second, third, and remaining skbs.

Right, as we handle the head skb above. Could you clarify?

Quoted 10 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5195

Quoted 4 lines
net/ipv4/tcp_input.c +1 -0
+                                       goto end;
 #endif
                        }
                }

tcp_collapse() is copying data from small skbs to pack it to bigger skb (one page of payload), in case
of memory emergency/pressure (socket queues are full)

Quoted 6 lines

If your changes are trying to avoid 'needless' copies, maybe you should reconsider and let the emergency packing be done.

If the copy is not _possible_, you should rephrase your changelog to clearly state the kernel _cannot_ access this memory in any way.

The issue is that skb_condense also gets called on many skbs in tcp_add_backlog and it will identify skbs that went through DDP as ideal for packing, even though they are not small and packing is counter-productive as data already resides in its destination.

As mentioned above, it is possible to copy, but it is counter-productive in this case. If there was a real need to access this memory, then it is allowed.

#4 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

Quoted 45 lines · Boris Pismenny
Quoted 9 lines · Eric Dumazet
Quoted 4 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

...

Quoted 2 lines

};

Quoted 16 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5154

Quoted 12 lines

const char

net/ipv4/tcp_input.c +3 -0
index e6ca5a1f3b59..4a7160bba09b 100644
@@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
                memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
                nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+               nskb->ddp_crc = skb->ddp_crc;

Probably you do not want to attempt any collapse if skb->ddp_crc is set right there.

Right.

Quoted 5 lines
Quoted 5 lines
net/ipv4/tcp_input.c +0 -0
 #endif
                TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
                if (list)
@@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
Quoted 9 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5191

Quoted 6 lines
net/ipv4/tcp_input.c +4 -0
                                if (skb->decrypted != nskb->decrypted)
                                        goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+
+                               if (skb->ddp_crc != nskb->ddp_crc)

This checks only the second, third, and remaining skbs.

Quoted 1 line

Right, as we handle the head skb above. Could you clarify?

I was simply saying you missed the first skb.

Quoted 18 lines
Quoted 11 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5195

Quoted 5 lines
net/ipv4/tcp_input.c +1 -0
+                                       goto end;
 #endif
                        }
                }

tcp_collapse() is copying data from small skbs to pack it to bigger skb (one page of payload), in case
of memory emergency/pressure (socket queues are full)

Quoted 6 lines

If your changes are trying to avoid 'needless' copies, maybe you should reconsider and let the emergency packing be done.

If the copy is not _possible_, you should rephrase your changelog to clearly state the kernel _cannot_ access this memory in any way.

Quoted 8 lines

The issue is that skb_condense also gets called on many skbs in tcp_add_backlog and it will identify skbs that went through DDP as ideal for packing, even though they are not small and packing is counter-productive as data already resides in its destination.

As mentioned above, it is possible to copy, but it is counter-productive in this case. If there was a real need to access this memory, then it is allowed.

Standard GRO packets from high perf drivers have no room in their skb->head (ie skb_tailroom() should be 0)

If you have a driver using GRO and who pulled some payload in skb->head, it is already too late for DDP.

So I think you are trying to add code in TCP that should not be needed. Perhaps mlx5 driver is doing something it should not ? (If this is ' copybreak' this has been documented as being suboptimal, transports have better strategies)

Secondly, tcp_collapse() should absolutely not be called under regular workloads.

Trying to optimize this last-resort thing is a lost cause:
If an application is dumb enough to send small packets back-to-back, it should be fixed (sender side has this thing called autocork, for applications that do not know about MSG_MORE or TC_CORK.)

(tcp_collapse is a severe source of latencies)

#9 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Boris Pismenny
Quoted 9 lines
include/linux/uio.h +3 -0
+#ifdef CONFIG_ULP_DDP
+size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
+#endif
 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i);
 bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i);
 size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
@@ -145,6 +148,16 @@ size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
 		return _copy_to_iter(addr, bytes, i);
 }

include/linux/uio.h size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) · line 159

Quoted 10 lines
include/linux/uio.h +9 -0
 
+#ifdef CONFIG_ULP_DDP
+static __always_inline __must_check
+size_t ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
+{
+	if (unlikely(!check_copy_size(addr, bytes, true)))
+		return 0;
+	return _ddp_copy_to_iter(addr, bytes, i);
+}
+#endif

There is no need to ifdef out externs with conditional implementations, or inlines using them.

lib/iov_iter.c void iov_iter_init(struct iov_iter *i, unsigned int direction, · line 512

Quoted 2 lines
lib/iov_iter.c +2 -0
+#ifdef CONFIG_ULP_DDP
+static void ddp_memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len)

Overly long line.

lib/iov_iter.c void iov_iter_init(struct iov_iter *i, unsigned int direction, · line 519

Quoted 6 lines
lib/iov_iter.c +6 -0
+	char *to = kmap_atomic(page);
+
+	if (to + offset != from)
+		memcpy(to + offset, from, len);
+
+	kunmap_atomic(to);

This looks completely bogus to any casual read, so please document why it makes sense. And no, a magic, unexplained ddp in the name does not count as explanation at all. Please think about a more useful name.

Can this ever write to user page? If yes it needs a flush_dcache_page.

Last but not least: kmap_atomic is deprecated except for the very rate use case where it is actually called from atomic context. Please use kmap_local_page instead.

lib/iov_iter.c size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, · line 1871

Quoted 13 lines
lib/iov_iter.c +13 -0
+#ifdef CONFIG_CRYPTO_HASH
+	struct ahash_request *hash = hashp;
+	struct scatterlist sg;
+	size_t copied;
+
+	copied = ddp_copy_to_iter(addr, bytes, i);
+	sg_init_one(&sg, addr, copied);
+	ahash_request_set_crypt(hash, &sg, NULL, copied);
+	crypto_ahash_update(hash);
+	return copied;
+#else
+	return 0;
+#endif

What is the point of this stub? To me it looks extremely dangerous.

#5 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

In reply to Eric Dumazet
Quoted 46 lines · Eric Dumazet
Quoted 45 lines · Boris Pismenny
Quoted 9 lines · Eric Dumazet
Quoted 4 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

...

Quoted 2 lines

};

Quoted 16 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5154

Quoted 12 lines

const char

net/ipv4/tcp_input.c +3 -0
index e6ca5a1f3b59..4a7160bba09b 100644
@@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
                memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
                nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+               nskb->ddp_crc = skb->ddp_crc;

Probably you do not want to attempt any collapse if skb->ddp_crc is set right there.

Right.

Quoted 14 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5191

Quoted 11 lines
net/ipv4/tcp_input.c +4 -0
 #endif
                TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
                if (list)
@@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
                                if (skb->decrypted != nskb->decrypted)
                                        goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+
+                               if (skb->ddp_crc != nskb->ddp_crc)

This checks only the second, third, and remaining skbs.

Quoted 4 lines
Quoted 1 line

Right, as we handle the head skb above. Could you clarify?

I was simply saying you missed the first skb.

But, the first SKB got handled in the change above. The code here is the same for TLS, if it is wrong, then we already have an issue here.

Quoted 41 lines
Quoted 18 lines
Quoted 11 lines

net/ipv4/tcp_input.c tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, · line 5195

Quoted 5 lines
net/ipv4/tcp_input.c +1 -0
+                                       goto end;
 #endif
                        }
                }

tcp_collapse() is copying data from small skbs to pack it to bigger skb (one page of payload), in case
of memory emergency/pressure (socket queues are full)

Quoted 6 lines

If your changes are trying to avoid 'needless' copies, maybe you should reconsider and let the emergency packing be done.

If the copy is not _possible_, you should rephrase your changelog to clearly state the kernel _cannot_ access this memory in any way.

Quoted 8 lines

The issue is that skb_condense also gets called on many skbs in tcp_add_backlog and it will identify skbs that went through DDP as ideal for packing, even though they are not small and packing is counter-productive as data already resides in its destination.

As mentioned above, it is possible to copy, but it is counter-productive in this case. If there was a real need to access this memory, then it is allowed.

Standard GRO packets from high perf drivers have no room in their skb->head (ie skb_tailroom() should be 0)

If you have a driver using GRO and who pulled some payload in skb->head, it is already too late for DDP.

So I think you are trying to add code in TCP that should not be needed. Perhaps mlx5 driver is doing something it should not ? (If this is ' copybreak' this has been documented as being suboptimal, transports have better strategies)

Secondly, tcp_collapse() should absolutely not be called under regular workloads.

Quoted 7 lines

Trying to optimize this last-resort thing is a lost cause:
If an application is dumb enough to send small packets back-to-back, it should be fixed (sender side has this thing called autocork, for applications that do not know about MSG_MORE or TC_CORK.)

(tcp_collapse is a severe source of latencies)

Sorry. My response above was about skb_condense which I've confused with tcp_collapse.

In tcp_collapse, we could allow the copy, but the problem is CRC, which like TLS's skb->decrypted marks that the data passed the digest validation in the NIC. If we allow collapsing SKBs with mixed marks, we will need to force software copy+crc verification. As TCP collapse is indeed rare and the offload is opportunistic in nature, we can make this change and submit another version, but I'm confused; why was it OK for TLS, while it is not OK for DDP+CRC?

#6 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

Quoted 11 lines · Boris Pismenny

Sorry. My response above was about skb_condense which I've confused with tcp_collapse.

In tcp_collapse, we could allow the copy, but the problem is CRC, which like TLS's skb->decrypted marks that the data passed the digest validation in the NIC. If we allow collapsing SKBs with mixed marks, we will need to force software copy+crc verification. As TCP collapse is indeed rare and the offload is opportunistic in nature, we can make this change and submit another version, but I'm confused; why was it OK for TLS, while it is not OK for DDP+CRC?

Ah.... I guess I was focused on the DDP part, while all your changes are really about the CRC part.

Perhaps having an accessor to express the CRC status (and not be confused by the DDP part)
could help the intent of the code.

#7 lore raw

Re: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload

Quoted 14 lines · Eric Dumazet
Quoted 11 lines · Boris Pismenny

Sorry. My response above was about skb_condense which I've confused with tcp_collapse.

In tcp_collapse, we could allow the copy, but the problem is CRC, which like TLS's skb->decrypted marks that the data passed the digest validation in the NIC. If we allow collapsing SKBs with mixed marks, we will need to force software copy+crc verification. As TCP collapse is indeed rare and the offload is opportunistic in nature, we can make this change and submit another version, but I'm confused; why was it OK for TLS, while it is not OK for DDP+CRC?

Quoted 7 lines

Ah.... I guess I was focused on the DDP part, while all your changes are really about the CRC part.

Perhaps having an accessor to express the CRC status (and not be confused by the DDP part)
could help the intent of the code.

An accessor function sounds like a great idea for readability, thanks Eric!

We will re-spin the series and add it to v6.

#45 lore raw

Re: [PATCH v5 net-next 24/36] net: Add MSG_DDP_CRC flag

In reply to Boris Pismenny
Quoted 6 lines · Boris Pismenny

From: Yoray Zack <[email protected]>

if the msg sent with this flag, turn up skb->ddp_crc bit.

Ok, but why ?

How would you document in linux manpages the purpose of this flag ?

How can applications be sure the kernel they run on is actually supporting this flag ?
Maybe it is a hint only ?

#10 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Christoph Hellwig
Quoted 28 lines · Christoph Hellwig
Quoted 9 lines
include/linux/uio.h +3 -0
+#ifdef CONFIG_ULP_DDP
+size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
+#endif
 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i);
 bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i);
 size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
@@ -145,6 +148,16 @@ size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
 		return _copy_to_iter(addr, bytes, i);
 }

include/linux/uio.h size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) · line 159

Quoted 10 lines
include/linux/uio.h +9 -0
 
+#ifdef CONFIG_ULP_DDP
+static __always_inline __must_check
+size_t ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
+{
+	if (unlikely(!check_copy_size(addr, bytes, true)))
+		return 0;
+	return _ddp_copy_to_iter(addr, bytes, i);
+}
+#endif

There is no need to ifdef out externs with conditional implementations, or inlines using them.

lib/iov_iter.c void iov_iter_init(struct iov_iter *i, unsigned int direction, · line 512

Quoted 2 lines
lib/iov_iter.c +2 -0
+#ifdef CONFIG_ULP_DDP
+static void ddp_memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len)

Overly long line.

Quoted 10 lines

lib/iov_iter.c void iov_iter_init(struct iov_iter *i, unsigned int direction, · line 519

Quoted 6 lines
lib/iov_iter.c +6 -0
+	char *to = kmap_atomic(page);
+
+	if (to + offset != from)
+		memcpy(to + offset, from, len);
+
+	kunmap_atomic(to);

This looks completely bogus to any casual read, so please document why it makes sense. And no, a magic, unexplained ddp in the name does not count as explanation at all. Please think about a more useful name.

This routine, like other changes in this file, replicates the logic in memcpy_to_page. The only difference is that "ddp" avoids copies when the copy source and destinations buffers are one and the same. These are then used by nvme-tcp (see skb_ddp_copy_datagram_iter in nvme-tcp) which receives SKBs from the NIC that already placed data in its destination, and this is the source for the name Direct Data Placement. I'd gladly take suggestions for better names, but this is the best we came up with so far.

The reason we are doing it is to avoid modifying memcpy_to_page itself, but rather allow users (e.g., nvme-tcp) to access this functionality directly.

Quoted 2 lines

Can this ever write to user page? If yes it needs a flush_dcache_page.

Yes, will add.

Quoted 5 lines

Last but not least: kmap_atomic is deprecated except for the very rate use case where it is actually called from atomic context. Please use kmap_local_page instead.

Will look into it, thanks!

Quoted 7 lines
Quoted 7 lines
lib/iov_iter.c +7 -0
+#ifdef CONFIG_CRYPTO_HASH
+	struct ahash_request *hash = hashp;
+	struct scatterlist sg;
+	size_t copied;
+
+	copied = ddp_copy_to_iter(addr, bytes, i);
+	sg_init_one(&sg, addr, copied);
Quoted 9 lines

lib/iov_iter.c size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, · line 1871

Quoted 6 lines
lib/iov_iter.c +6 -0
+	ahash_request_set_crypt(hash, &sg, NULL, copied);
+	crypto_ahash_update(hash);
+	return copied;
+#else
+	return 0;
+#endif

What is the point of this stub? To me it looks extremely dangerous.

As above, we use the same logic as in hash_and_copy_to_iter. The purpose is again to eventually avoid the copy in case the source and destination buffers are one and the same.

maintainer
#15 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Boris Pismenny
Quoted 11 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

When using direct data placement (DDP) the NIC writes some of the payload directly to the destination buffer, and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination we use the newly introduced routines in this commit, which check if (src == dst), and skip the copy when that's true.

As the current user for these routines is in the block layer (nvme-tcp), then we only apply the change for bio_vec. Other routines use the normal methods for copying.

Please, take a look at -rc1 and see the changes in lib/iov_iter.c in there.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#11 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Boris Pismenny
Quoted 3 lines · Boris Pismenny

This routine, like other changes in this file, replicates the logic in memcpy_to_page. The only difference is that "ddp" avoids copies when the copy source and destinations buffers are one and the same.

Now why can't we just make that change to the generic routine?

If we can't, why do they not have a saner name documenting what they actually do?

maintainer
#12 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Boris Pismenny

This routine, like other changes in this file, replicates the logic in memcpy_to_page. The only difference is that "ddp" avoids copies when the copy source and destinations buffers are one and the same.

Now why can't we just make that change to the generic routine?

Doable... replace memcpy(base, addr + off, len) with

	base != addr + off && memcpy(base, addr + off, len)

in _copy_to_iter() and be done with that...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#58 lore raw
In reply to Boris Pismenny
Quoted 4 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

Changes since v4:
=========================================

Explaining what the series does should go before the changelog.

Quoted 1 line

* Add transmit offload patches

But to be honest, the previous one was already mostly unreviewable,
but this is now far beyond this. Please try to get anything that
is generally useful first in smaller series and the come back with a somewhat reviewable series. That also means that at least for the code I care about (nvme) the patches should be grouped together, and actually provide meaningful functionality in each patch. Right now even trying to understand what you add to the nvme code requires me to jump all over a gigantic series.

#18 lore raw

Re: [PATCH v5 net-next 04/36] net/tls: expose get_netdev_for_sock

In reply to Boris Pismenny
Quoted 9 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.

I don't think this should be an inline. Please move it to net/core/dev.c, andd add an EXPORT_SYMBOL_GPL and a kerneldoc comment.

#43 lore raw

Re: [PATCH v5 net-next 23/36] net: Add to ulp_ddp support for fallback flow

In reply to Boris Pismenny

include/net/ulp_ddp.h struct ulp_ddp_ulp_ops { · line 135

Quoted 4 lines · Boris Pismenny
include/net/ulp_ddp.h +2 -0
 	/* NIC driver informs the ulp that ddp teardown is done - used for async completions*/
 	void (*ddp_teardown_done)(void *ddp_ctx);
+	/* NIC request ulp to calculate the ddgst and store it in pdu_info->ddgst */
+	void (*ddp_ddgst_fallback)(struct ulp_ddp_pdu_info *pdu_info);

Overly long line. More importantly this whole struct should probably use a kerneldoc comment anyway.

net/core/ulp_ddp.c struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq) · line 197

Quoted 3 lines

} EXPORT_SYMBOL(ulp_ddp_get_pdu_info);

net/core/ulp_ddp.c +1 -0
+	if (!pdu_info || !between(seq, pdu_info->start_seq, pdu_info->end_seq - 1)) {

More overly lone lines. Please make sure to stick to 80 character lines unless you have a really good to go over that.

net/core/ulp_ddp.c struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq) · line 205

Quoted 1 line
net/core/ulp_ddp.c +1 -0
+	//check if this skb contains ddgst field

Plase avoid //-style comments.

net/core/ulp_ddp.c struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq) · line 235

Quoted 2 lines
net/core/ulp_ddp.c +2 -0
+	return ulp_ddp_fallback_skb(ctx, skb, sk);
+} EXPORT_SYMBOL(ulp_ddp_validate_xmit_skb);

This is not how EXPORT_SYMBOLs are place. Also please export any such deep internal interfaces using EXPORT_SYMBOL_GPL.

maintainer
#41 lore raw

Re: [PATCH v5 net-next 22/36] net: Add ulp_ddp_pdu_info struct

In reply to Boris Pismenny
Quoted 9 lines
include/net/ulp_ddp.h +8 -0
+/**
+ * struct ulp_ddp_pdu_info - pdu info for tcp ddp crc Tx offload.
+ *
+ * @end_seq:	tcp seq of the last byte in the pdu.
+ * @start_seq:	tcp seq of the first byte in the pdu.
+ * @data_len:	pdu data size (in bytes).
+ * @hdr_len:	the size (in bytes) of the pdu header.
+ * @hdr:	pdu header.

include/net/ulp_ddp.h struct ulp_ddp_io { · line 88

Quoted 10 lines
include/net/ulp_ddp.h +10 -0
+ * @req:	the ulp request for the original pdu.
+ */
+struct ulp_ddp_pdu_info {
+	struct list_head list;
+	u32		end_seq;
+	u32		start_seq;
+	u32		data_len;
+	u32		hdr_len;
+	void		*hdr;
+	struct request	*req;

Not sure what ddp does with this, but it shouldn't accept struct request what-so-ever.

maintainer
#59 lore raw
In reply to Christoph Hellwig
Quoted 11 lines
Quoted 2 lines

* Add transmit offload patches

But to be honest, the previous one was already mostly unreviewable,
but this is now far beyond this. Please try to get anything that
is generally useful first in smaller series and the come back with a somewhat reviewable series. That also means that at least for the code I care about (nvme) the patches should be grouped together, and actually provide meaningful functionality in each patch. Right now even trying to understand what you add to the nvme code requires me to jump all over a gigantic series.

I agree as well. It is difficult to review.
The order should be:
1. ulp_ddp interface
2. nvme-tcp changes
3. mlx5e changes

Also even beyond grouping patches together I have 2 requests:
1. Please consolidate ddp routines under a single ifdef (also minimize the ifdef in call-sites).
2. When consolidating functions, try to do this as prep patches documenting in the change log that it is preparing to add ddp. Its difficult digesting both at times.

#19 lore raw

Re: [PATCH v5 net-next 04/36] net/tls: expose get_netdev_for_sock

In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Quoted 7 lines
Quoted 4 lines

Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.

I don't think this should be an inline. Please move it to net/core/dev.c, andd add an EXPORT_SYMBOL_GPL and a kerneldoc comment.

Jakub,

What's your preference here?

Or.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#60 lore raw
In reply to Sagi Grimberg
Quoted 5 lines · Sagi Grimberg

[.. ] It is difficult to review.
The order should be:
1. ulp_ddp interface
2. nvme-tcp changes
3. mlx5e changes

.. and this is exactly how the series is organized, for v6 we will drop the TX offload part and stick to completing the review on the RX offload part.

Quoted 3 lines

Also even beyond grouping patches together I have 2 requests:
1. Please consolidate ddp routines under a single ifdef (also minimize the ifdef in call-sites).

ok, will make an effort to be better in that respect

Quoted 3 lines

2. When consolidating functions, try to do this as prep patches documenting in the change log that it is preparing to add ddp. Its difficult digesting both at times.

to clarify, you would like patch #5 "nvme-tcp: Add DDP offload control path" to only add the call sites and if-not-deffed implementation for the added knobs:

nvme_tcp_offload_socket
nvme_tcp_unoffload_socket
nvme_tcp_offload_limits
nvme_tcp_resync_response

and a 2nd patch to add the if-yes-deffed implementation?

This makes sense, however IMHO repeating this prep exercise for the data-path patch (#6 "nvme-tcp: Add DDP data-path") doesn't seem to provide notable value b/c you will only see two call sites for the two added empty knobs:

nvme_tcp_setup_ddp
nvme_tcp_teardown_ddp

but whatever you prefer, so.. let us know

#13 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Al Viro
Quoted 7 lines · Al Viro
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Boris Pismenny

This routine, like other changes in this file, replicates the logic in memcpy_to_page. The only difference is that "ddp" avoids copies when the copy source and destinations buffers are one and the same.

Quoted 5 lines
Quoted 1 line

Now why can't we just make that change to the generic routine?

Doable... replace memcpy(base, addr + off, len) with

        base != addr + off && memcpy(base, addr + off, len)

in _copy_to_iter() and be done with that...

Guys,

AFAIR we did the adding ddp_ prefix exercise to the copy functions call chain

ddp_hash_and_copy_to_iter
-> ddp_copy_to_iter
-> _ddp_copy_to_iter
-> ddp_memcpy_to_page

to address feedback given on earlier versions of the series. So let's decide please.. are we all set to remove the ddp_ prefixed calls and just plant the new check (plus a nice comment!) as Al suggested?

re the comments given on ddp_memcpy_to_page, upstream move to just call memcpy, so we need not have it anyway, will be fixed in v6 if we remain with ddp_ call chain or becomes irrelevant if we drop it.

#21 lore raw

Re: [PATCH v5 net-next 04/36] net/tls: expose get_netdev_for_sock

In reply to a message not in the archive
Quoted 14 lines · Jakub Kicinski
Quoted 12 lines · On Wed, 4 Aug 2021 16:26:31 +0300 Or Gerlitz wrote:
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I don't think this should be an inline. Please move it to net/core/dev.c, andd add an EXPORT_SYMBOL_GPL and a kerneldoc comment.

Jakub,

What's your preference here?

Fine either way.

copying the list and few ppl

maintainer
#61 lore raw
In reply to Or Gerlitz
Quoted 36 lines · Or Gerlitz
Quoted 5 lines · Sagi Grimberg

[.. ] It is difficult to review.
The order should be:
1. ulp_ddp interface
2. nvme-tcp changes
3. mlx5e changes

.. and this is exactly how the series is organized, for v6 we will drop the TX offload part and stick to completing the review on the RX offload part.

Quoted 3 lines

Also even beyond grouping patches together I have 2 requests:
1. Please consolidate ddp routines under a single ifdef (also minimize the ifdef in call-sites).

ok, will make an effort to be better in that respect

Quoted 3 lines

2. When consolidating functions, try to do this as prep patches documenting in the change log that it is preparing to add ddp. Its difficult digesting both at times.

to clarify, you would like patch #5 "nvme-tcp: Add DDP offload control path" to only add the call sites and if-not-deffed implementation for the added knobs:

nvme_tcp_offload_socket
nvme_tcp_unoffload_socket
nvme_tcp_offload_limits
nvme_tcp_resync_response

and a 2nd patch to add the if-yes-deffed implementation?

This makes sense, however IMHO repeating this prep exercise for the data-path patch (#6 "nvme-tcp: Add DDP data-path") doesn't seem to provide notable value b/c you will only see two call sites for the two added empty knobs:

Quoted 4 lines

nvme_tcp_setup_ddp
nvme_tcp_teardown_ddp

but whatever you prefer, so.. let us know

I was more referring to routines that now grew the ddp path and changed in the same time like:
nvme_tcp_complete_request
nvme_tcp_consume_skb
etc..

#22 lore raw

Re: [PATCH v5 net-next 04/36] net/tls: expose get_netdev_for_sock

In reply to Or Gerlitz
Quoted 17 lines · Or Gerlitz
Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Wed, 4 Aug 2021 16:26:31 +0300 Or Gerlitz wrote:
Quoted 13 lines · Christoph Hellwig
Quoted 9 lines · Boris Pismenny

From: Boris Pismenny <[email protected]>

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.

I don't think this should be an inline. Please move it to net/core/dev.c, andd add an EXPORT_SYMBOL_GPL and a kerneldoc comment.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Jakub,
What's your preference here?

Fine either way.

copying the list and few ppl

Christoph,

We will add the kdoc.

Prefer to leave it as inline which is fine by the networking maintainer.

Or.

#14 lore raw

Re: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages

In reply to Or Gerlitz
Quoted 18 lines · Or Gerlitz
Quoted 7 lines · Al Viro
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Boris Pismenny

This routine, like other changes in this file, replicates the logic in memcpy_to_page. The only difference is that "ddp" avoids copies when the copy source and destinations buffers are one and the same.

Quoted 5 lines
Quoted 1 line

Now why can't we just make that change to the generic routine?

Doable... replace memcpy(base, addr + off, len) with

        base != addr + off && memcpy(base, addr + off, len)

in _copy_to_iter() and be done with that...

Guys,

AFAIR we did the adding ddp_ prefix exercise to the copy functions call chain

Quoted 8 lines

ddp_hash_and_copy_to_iter
-> ddp_copy_to_iter
-> _ddp_copy_to_iter
-> ddp_memcpy_to_page

to address feedback given on earlier versions of the series. So let's decide please.. are we all set to remove the ddp_ prefixed calls and just plant the new check (plus a nice comment!) as Al suggested?

So we are okay going for the minimal approach / direction suggested by Al of adding a (base != addr + offset) check before the memcpy call.

This will also simplify the changes to the nvme-tcp driver. Please speak if you want the ddp_ prefix approach to remain.

Or.

Quoted 3 lines

re the comments given on ddp_memcpy_to_page, upstream move to just call memcpy, so we need not have it anyway, will be fixed in v6 if we remain with ddp_ call chain or becomes irrelevant if we drop it.

#62 lore raw
In reply to Sagi Grimberg
Quoted 40 lines · Sagi Grimberg
Quoted 39 lines · Or Gerlitz
Quoted 5 lines · Sagi Grimberg

[.. ] It is difficult to review.
The order should be:
1. ulp_ddp interface
2. nvme-tcp changes
3. mlx5e changes

.. and this is exactly how the series is organized, for v6 we will drop the TX offload part and stick to completing the review on the RX offload part.

Quoted 3 lines

Also even beyond grouping patches together I have 2 requests:
1. Please consolidate ddp routines under a single ifdef (also minimize the ifdef in call-sites).

ok, will make an effort to be better in that respect

Quoted 3 lines

2. When consolidating functions, try to do this as prep patches documenting in the change log that it is preparing to add ddp. Its difficult digesting both at times.

to clarify, you would like patch #5 "nvme-tcp: Add DDP offload control path" to only add the call sites and if-not-deffed implementation for the added knobs:

nvme_tcp_offload_socket
nvme_tcp_unoffload_socket
nvme_tcp_offload_limits
nvme_tcp_resync_response

and a 2nd patch to add the if-yes-deffed implementation?

This makes sense, however IMHO repeating this prep exercise for the data-path patch (#6 "nvme-tcp: Add DDP data-path") doesn't seem to provide notable value b/c you will only see two call sites for the two added empty knobs:

nvme_tcp_setup_ddp
nvme_tcp_teardown_ddp

Quoted 6 lines
Quoted 1 line

but whatever you prefer, so.. let us know

I was more referring to routines that now grew the ddp path and changed in the same time like:

nvme_tcp_complete_request

not sure to follow on this one.. It's added on patch #6 "nvme-tcp: Add DDP data-path"
and then used twice in the same patch replacing calls to nvme_try_complete_req and then to nvme_complete_rq -- so how want this be broken to prep/usage?

Quoted 1 line

nvme_tcp_consume_skb

this routine was born due to the ddp_ prefix addition to the iov copy iter functions, which we are now removing due to feedback from Al

Quoted 1 line

etc..