Skip to contentSkip to conversation

nvme-tcp receive offloads

RFC PATCH In review Boris Pismenny 6 participants 38 messages Sep 30, 2020 – Nov 11, 2020 3 messages not in the archive

Subsystems (13)
lore
Share & tools
lore
#0 lore raw

cover nvme-tcp receive offloads

This series adds support for nvme-tcp receive offloads which do not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers
2. CRC verification for received PDU

The series implements these as a generic offload infrastructure for storage protocols, which we call TCP Direct Data Placement (TCP_DDP) and TCP DDP CRC, respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC. Future implementations can reuse the same infrastructure for other protcols such as iSCSI.

Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html

Initialization and teardown:
=========================================
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by calling nvme_tcp_offload_socket with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usually with no offload. If it succeeds then nvme_tcp_setup_ddp and nvme_tcp_teardown_ddp may be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of nvme-tcp in any way besides adding the option to call the above mentioned NDO operations.

For the admin queue, nvme-tcp does not initialize the offload. Instead, nvme-tcp calls the driver to configure limits for the controller, such as max_hw_sectors and max_segments; these must be limited to accomodate potential HW resource limits, and to improve performance.

If some error occured, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.

Copy offload works as follows:
=========================================
The nvme-tcp layer calls the NIC drive to map block layer buffers to ccid using nvme_tcp_setup_ddp before sending the read request. When the repsonse is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer; this SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once nvme-tcp attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function (memcpy_to_page):
if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the nvme-tcp layer releases the NIC HW context be calling nvme_tcp_teardown_ddp which asynchronously unmaps the buffers from NIC HW.

As the last change is to a sensative function, we are careful to place it under static_key which is only enabled when this functionality is actually used for nvme-tcp copy offload.

Asynchronous completion:
=========================================
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance criticial, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).

An alternative approach is to move all the functions related to coping from SKBs to the block layer buffers inside the nvme-tcp code - about 200 LOC.

CRC offload works as follows:
=========================================
After offload is initialized, we use the SKB's ddp_crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (ddp_crc != 1), then software must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.

SKB changes:
=========================================
The CRC offload requires an additional bit in the SKB, which is useful for preventing the coalescing of SKB with different crc offload values. This bit is similar in concept to the "decrypted" bit.

Performance:
=========================================
The expected performance gain from this offload varies with the block size. We perform a CPU cycles breakdown of the copy/CRC operations in nvme-tcp fio random read workloads:
For 4K blocks we see up to 11% improvement for a 100% read fio workload, while for 128K blocks we see upto 52%. If we run nvme-tcp, and skip these operations, then we observe a gain of about 1.1x and 2x respectively.

Resynchronization:
=========================================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the driver, regarding a possible location of a PDU header. Followed by a response from the nvme-tcp driver.

This flow is rare, and it should happen only after packet loss or reordering events that involve nvme-tcp PDU headers.

The patches are organized as follows:
=========================================
Patch 1 the iov_iter change to skip copy if (src == dst)
Patches 2-3 the infrastructure for all TCP DDP
and TCP DDP CRC offloads, respectively.

Patch 4         exposes the get_netdev_for_sock function from TLS
Patch 5         NVMe-TCP changes to call NIC driver on queue init/teardown
Patches 6       NVMe-TCP changes to call NIC driver on IO operation

setup/teardown, and support async completions.
Patches 7 NVMe-TCP changes to support CRC offload on receive.
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patches 8 NVMe-TCP handling of netdev events: stop the offload if
netdev is going down
Patches 9-10 implement support for NVMe-TCP copy and CRC offload in
the mlx5 NIC driver

Testing:
=========================================
This series was tested using fio with various configurations of IO sizes, depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets.

Future work:
=========================================
A follow-up series will introduce support for transmit side CRC. Then, we will work on adding support for TLS in NVMe-TCP and combining the two offloads.

Boris Pismenny (8):
iov_iter: Skip copy in memcpy_to_page if src==dst
net: Introduce direct data placement tcp offload
net: Introduce crc offload for tcp ddp ulp net/tls: expose get_netdev_for_sock
nvme-tcp: Add DDP offload control path
nvme-tcp: Add DDP data-path
net/mlx5e: Add NVMEoTCP offload net/mlx5e: NVMEoTCP, data-path for DDP offload

Or Gerlitz (1):
nvme-tcp: Deal with netdevice DOWN events

Yoray Zack (1):
nvme-tcp : Recalculate crc in the end of the capsule

 .../net/ethernet/mellanox/mlx5/core/Kconfig   |  11 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  12 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |   4 +
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |  13 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |   1 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |   9 +-
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      |  10 +
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 894 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    | 116 +++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        | 256 +++++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.h        |  25 +
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |  79 ++
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  73 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  76 +-
 .../ethernet/mellanox/mlx5/core/en_stats.c    |  38 +
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  24 +
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |   6 +
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   6 +
 drivers/nvme/host/tcp.c                       | 410 +++++++-
 include/linux/mlx5/device.h                   |  38 +-
 include/linux/mlx5/mlx5_ifc.h                 | 121 ++-
 include/linux/mlx5/qp.h                       |   1 +
 include/linux/netdev_features.h               |   4 +
 include/linux/netdevice.h                     |   5 +
 include/linux/nvme-tcp.h                      |   2 +
 include/linux/skbuff.h                        |   4 +
 include/linux/uio.h                           |   2 +
 include/net/inet_connection_sock.h            |   4 +
 include/net/sock.h                            |  17 +
 include/net/tcp_ddp.h                         |  90 ++
 lib/iov_iter.c                                |  11 +-
 net/Kconfig                                   |  17 +
 net/core/skbuff.c                             |   9 +-
 net/ethtool/common.c                          |   2 +
 net/ipv4/tcp_input.c                          |   7 +
 net/ipv4/tcp_ipv4.c                           |   3 +
 net/ipv4/tcp_offload.c                        |   3 +
 net/tls/tls_device.c                          |  20 +-
 40 files changed, 2375 insertions(+), 51 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
 create mode 100644 include/net/tcp_ddp.h

base-commit: 300fd579b2e8608586b002207e906ac95c74b911
prerequisite-patch-id: b3079fa1f4c0e3d6d4a92f59e70981e8a28f3b83

Signature · 1 line
2.24.1
#1 lore raw

RFC PATCH 1/10 iov_iter: Skip copy in memcpy_to_page if src==dst

When using direct data placement the NIC writes some of the payload directly to the destination buffer, and constructs the SKB such that it points to this data. As a result, the skb_copy datagram_iter call will attempt to copy data when it is not necessary.

This patch adds a check to avoid this copy, and a static_key to enabled it when TCP direct data placement is possible.

---
 include/linux/uio.h |  2 ++
 lib/iov_iter.c      | 11 ++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
include/linux/uio.h +2 -0
index 3835a8a8e9ea..036df4073737 100644
@@ -284,4 +284,6 @@ int iov_iter_for_each_range(struct iov_iter *i, size_t bytes,
 			    int (*f)(struct kvec *vec, void *context),
 			    void *context);
 
+extern struct static_key_false skip_copy_enabled;
+
 #endif
lib/iov_iter.c +10 -1
index 5e40786c8f12..27856c3db28b 100644
@@ -13,6 +13,9 @@
 
 #define PIPE_PARANOIA /* for now */
 
+DEFINE_STATIC_KEY_FALSE(skip_copy_enabled);
+EXPORT_SYMBOL_GPL(skip_copy_enabled);
+
 #define iterate_iovec(i, n, __v, __p, skip, STEP) {	\
 	size_t left;					\
 	size_t wanted = n;				\
@@ -470,7 +473,13 @@ static void memcpy_from_page(char *to, struct page *page, size_t offset, size_t
 static void memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len)
 {
 	char *to = kmap_atomic(page);
-	memcpy(to + offset, from, len);
+
+	if (static_branch_unlikely(&skip_copy_enabled)) {
+		if (to + offset != from)
+			memcpy(to + offset, from, len);
+	} else {
+		memcpy(to + offset, from, len);
+	}
 	kunmap_atomic(to);
 }
 
Signature · 1 line
2.24.1
maintainer
#2 lore raw

Re: [PATCH net-next RFC v1 01/10] iov_iter: Skip copy in memcpy_to_page if src==dst

You probably want Al to have a look at this..

#3 lore raw

RFC PATCH 2/10 net: Introduce direct data placement tcp offload

In reply to Boris Pismenny

This commit introduces direct data placement offload for TCP. This capability is accompanied by new net_device operations that configure
hardware contexts. There is a context per socket, and a context per DDP opreation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments.

Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.

This, the SKB represents the data on the wire, while it is pointing to data that is already placed in the destination buffer. As a result, data from page frags should not be copied out to the linear part.

As SKBs that use DDP are already very memory efficient, we modify skb_condence to avoid copying data from fragments to the linear part of SKBs that belong to a socket that uses DDP offload.

A follow-up patch will use this interface for DDP in NVMe-TCP.

---
 include/linux/netdev_features.h    |  2 +
 include/linux/netdevice.h          |  5 ++
 include/net/inet_connection_sock.h |  4 ++
 include/net/tcp_ddp.h              | 90 ++++++++++++++++++++++++++++++
 net/Kconfig                        |  9 +++
 net/core/skbuff.c                  |  9 ++-
 net/ethtool/common.c               |  1 +
 7 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 include/net/tcp_ddp.h
include/linux/netdev_features.h +2 -0
index 0b17c4322b09..a0074b244372 100644
@@ -84,6 +84,7 @@ enum {
 	NETIF_F_GRO_FRAGLIST_BIT,	/* Fraglist GRO */
 
 	NETIF_F_HW_MACSEC_BIT,		/* Offload MACsec operations */
+	NETIF_F_HW_TCP_DDP_BIT,		/* TCP direct data placement offload */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -157,6 +158,7 @@ enum {
 #define NETIF_F_GRO_FRAGLIST	__NETIF_F(GRO_FRAGLIST)
 #define NETIF_F_GSO_FRAGLIST	__NETIF_F(GSO_FRAGLIST)
 #define NETIF_F_HW_MACSEC	__NETIF_F(HW_MACSEC)
+#define NETIF_F_HW_TCP_DDP	__NETIF_F(HW_TCP_DDP)
 
 /* Finds the next feature with the highest number of the range of start till 0.
  */
include/linux/netdevice.h +5 -0
index a431c3229cbf..b00e1663724b 100644
@@ -935,6 +935,7 @@ struct dev_ifalias {
 
 struct devlink;
 struct tlsdev_ops;
+struct tcp_ddp_dev_ops;
 
 struct netdev_name_node {
 	struct hlist_node hlist;
@@ -1922,6 +1923,10 @@ struct net_device {
 	const struct tlsdev_ops *tlsdev_ops;
 #endif
 
+#ifdef CONFIG_TCP_DDP
+	const struct tcp_ddp_dev_ops *tcp_ddp_ops;
+#endif
+
 	const struct header_ops *header_ops;
 
 	unsigned int		flags;
include/net/inet_connection_sock.h +4 -0
index dc763ca9413c..503092d8860d 100644
@@ -66,6 +66,8 @@ struct inet_connection_sock_af_ops {
  * @icsk_ulp_ops	   Pluggable ULP control hook
  * @icsk_ulp_data	   ULP private data
  * @icsk_clean_acked	   Clean acked data hook
+ * @icsk_ulp_ddp_ops	   Pluggable ULP direct data placement control hook
+ * @icsk_ulp_ddp_data	   ULP direct data placement private data
  * @icsk_listen_portaddr_node	hash to the portaddr listener hashtable
  * @icsk_ca_state:	   Congestion control state
  * @icsk_retransmits:	   Number of unrecovered [RTO] timeouts
@@ -94,6 +96,8 @@ struct inet_connection_sock {
 	const struct tcp_ulp_ops  *icsk_ulp_ops;
 	void __rcu		  *icsk_ulp_data;
 	void (*icsk_clean_acked)(struct sock *sk, u32 acked_seq);
+	const struct tcp_ddp_ulp_ops  *icsk_ulp_ddp_ops;
+	void __rcu		  *icsk_ulp_ddp_data;
 	struct hlist_node         icsk_listen_portaddr_node;
 	unsigned int		  (*icsk_sync_mss)(struct sock *sk, u32 pmtu);
 	__u8			  icsk_ca_state:5,
include/net/tcp_ddp.h new file +90 -0

90 lines not on this page · raw message

net/Kconfig +9 -0
index 3831206977a1..346e9fa7a6ec 100644
@@ -460,6 +460,15 @@ config ETHTOOL_NETLINK
 	  netlink. It provides better extensibility and some new features,
 	  e.g. notification messages.
 
+config TCP_DDP
+	bool "TCP direct data placement offload"
+	default n
+	help
+	  Direct Data Placement (DDP) offload for TCP enables ULP, such as
+	  NVMe-TCP/iSCSI, to request the NIC to place TCP payload data
+	  of a command response directly into kernel pages.
+
+
 endif   # if NET
 
 # Used by archs to tell that they support BPF JIT compiler plus which flavour.
net/core/skbuff.c +8 -1
index e0774471f56d..ad32985876da 100644
@@ -69,6 +69,7 @@
 #include <net/xfrm.h>
 #include <net/mpls.h>
 #include <net/mptcp.h>
+#include <net/tcp_ddp.h>
 
 #include <linux/uaccess.h>
 #include <trace/events/skb.h>
@@ -6059,9 +6060,15 @@ EXPORT_SYMBOL(pskb_extract);
  */
 void skb_condense(struct sk_buff *skb)
 {
+	bool is_ddp = false;
+
+#ifdef CONFIG_TCP_DDP
+	is_ddp = skb->sk && inet_csk(skb->sk) &&
+		 inet_csk(skb->sk)->icsk_ulp_ddp_data;
+#endif
 	if (skb->data_len) {
 		if (skb->data_len > skb->end - skb->tail ||
-		    skb_cloned(skb))
+		    skb_cloned(skb) || is_ddp)
 			return;
 
 		/* Nice, we can free page frag(s) right now */
net/ethtool/common.c +1 -0
index 24036e3055a1..a2ff7a4a6bbf 100644
@@ -68,6 +68,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {
 	[NETIF_F_HW_TLS_RX_BIT] =	 "tls-hw-rx-offload",
 	[NETIF_F_GRO_FRAGLIST_BIT] =	 "rx-gro-list",
 	[NETIF_F_HW_MACSEC_BIT] =	 "macsec-hw-offload",
+	[NETIF_F_HW_TCP_DDP_BIT] =	 "tcp-ddp-offload",
 };
 
 const char
Signature · 1 line
2.24.1
maintainer
#4 lore raw

Re: [PATCH net-next RFC v1 02/10] net: Introduce direct data placement tcp offload

include/net/tcp_ddp.h line 10

Quoted 9 lines

+ * tcp_ddp.h
+ * Author: Boris Pismenny <[email protected]>
+ * Copyright (C) 2020 Mellanox Technologies.
+ */

include/net/tcp_ddp.h new file +4 -0
+#ifndef _TCP_DDP_H
+#define _TCP_DDP_H
+
+#include <linux/blkdev.h>

Why is blkdev.h needed?

Quoted 6 lines
include/net/tcp_ddp.h new file +6 -0
+#include <linux/netdevice.h>
+#include <net/inet_connection_sock.h>
+#include <net/sock.h>
+
+/* limits returned by the offload driver, zero means don't care */
+struct tcp_ddp_limits {

include/net/tcp_ddp.h line 26

Quoted 10 lines
include/net/tcp_ddp.h new file +10 -0
+	int	 max_ddp_sgl_len;
+};
+
+enum tcp_ddp_type {
+	TCP_DDP_NVME = 1,
+};
+
+struct tcp_ddp_config {
+	enum tcp_ddp_type    type;
+	unsigned char        buf[];

A little kdoc may help here as its not exactly clear what is buf used for (at this point at least)...

include/net/tcp_ddp.h line 29

Quoted 3 lines
include/net/tcp_ddp.h new file +3 -0
+};
+
+struct nvme_tcp_config {

struct nvme_tcp_ddp_config

include/net/tcp_ddp.h line 39

Quoted 10 lines
include/net/tcp_ddp.h new file +10 -0
+	struct tcp_ddp_config   cfg;
+
+	u16			pfv;
+	u8			cpda;
+	u8			dgst;
+	int			queue_size;
+	int			queue_id;
+	int			io_cpu;
+};
+

Other than that this looks good to me.

#5 lore raw

Re: [PATCH net-next RFC v1 02/10] net: Introduce direct data placement tcp offload

Quoted 9 lines · Sagi Grimberg

include/net/tcp_ddp.h line 10

Quoted 8 lines

+ * tcp_ddp.h
+ * Author: Boris Pismenny <[email protected]>
+ * Copyright (C) 2020 Mellanox Technologies.
+ */

include/net/tcp_ddp.h new file +4 -0
+#ifndef _TCP_DDP_H
+#define _TCP_DDP_H
+
+#include <linux/blkdev.h>

Why is blkdev.h needed?

That's a lefotover from a previous iteration over this code. I'll remove it for the next patch.

Quoted 11 lines
Quoted 11 lines
include/net/tcp_ddp.h new file +10 -0
+#include <linux/netdevice.h>
+#include <net/inet_connection_sock.h>
+#include <net/sock.h>
+
+/* limits returned by the offload driver, zero means don't care */
+struct tcp_ddp_limits {
+	int	 max_ddp_sgl_len;
+};
+
+enum tcp_ddp_type {
Quoted 8 lines

include/net/tcp_ddp.h line 26

Quoted 6 lines
include/net/tcp_ddp.h new file +6 -0
+	TCP_DDP_NVME = 1,
+};
+
+struct tcp_ddp_config {
+	enum tcp_ddp_type    type;
+	unsigned char        buf[];

A little kdoc may help here as its not exactly clear what is buf used for (at this point at least)...

Will add.

Quoted 4 lines

include/net/tcp_ddp.h line 29

Quoted 3 lines
include/net/tcp_ddp.h new file +3 -0
+};
+
+struct nvme_tcp_config {

struct nvme_tcp_ddp_config

Sure.

Quoted 11 lines

include/net/tcp_ddp.h line 39

Quoted 10 lines
include/net/tcp_ddp.h new file +10 -0
+	struct tcp_ddp_config   cfg;
+
+	u16			pfv;
+	u8			cpda;
+	u8			dgst;
+	int			queue_size;
+	int			queue_id;
+	int			io_cpu;
+};
+

Other than that this looks good to me.

Thanks Sagi!

#6 lore raw

RFC PATCH 3/10 net: Introduce crc offload for tcp ddp ulp

In reply to Boris Pismenny

This commit itroduces support for CRC offload to direct data placement ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.

On the receive side, CRC offload requires a new SKB bit that indicates that no CRC error was encountered while processing this packet. If all packets of a ULP message have this bit set, then the CRC verification for the message can be skipped, as hardware already checked it.

The following patches will set and use this bit to perform NVME-TCP CRC offload.

A subsequent series, will add NVMe-TCP transmit side CRC support.

---
 include/linux/netdev_features.h | 2 ++
 include/linux/skbuff.h          | 4 ++++
 net/Kconfig                     | 8 ++++++++
 net/ethtool/common.c            | 1 +
 net/ipv4/tcp_input.c            | 7 +++++++
 net/ipv4/tcp_ipv4.c             | 3 +++
 net/ipv4/tcp_offload.c          | 3 +++
 7 files changed, 28 insertions(+)
include/linux/netdev_features.h +2 -0
index a0074b244372..27001f5c0be1 100644
@@ -85,6 +85,7 @@ enum {
 
 	NETIF_F_HW_MACSEC_BIT,		/* Offload MACsec operations */
 	NETIF_F_HW_TCP_DDP_BIT,		/* TCP direct data placement offload */
+	NETIF_F_HW_TCP_DDP_CRC_BIT,	/* TCP DDP CRC offload */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -159,6 +160,7 @@ enum {
 #define NETIF_F_GSO_FRAGLIST	__NETIF_F(GSO_FRAGLIST)
 #define NETIF_F_HW_MACSEC	__NETIF_F(HW_MACSEC)
 #define NETIF_F_HW_TCP_DDP	__NETIF_F(HW_TCP_DDP)
+#define NETIF_F_HW_TCP_DDP_CRC	__NETIF_F(HW_TCP_DDP_CRC)
 
 /* Finds the next feature with the highest number of the range of start till 0.
  */
include/linux/skbuff.h +4 -0
index 04a18e01b362..0530d849ebf2 100644
@@ -858,6 +858,10 @@ struct sk_buff {
 #ifdef CONFIG_TLS_DEVICE
 	__u8			decrypted:1;
 #endif
+#ifdef CONFIG_TCP_DDP_CRC
+	__u8                    ddp_crc:1;
+#endif
+
 
 #ifdef CONFIG_NET_SCHED
 	__u16			tc_index;	/* traffic control index */
net/Kconfig +8 -0
index 346e9fa7a6ec..7891b4380e56 100644
@@ -468,6 +468,14 @@ config TCP_DDP
 	  NVMe-TCP/iSCSI, to request the NIC to place TCP payload data
 	  of a command response directly into kernel pages.
 
+config TCP_DDP_CRC
+	bool "TCP direct data placement CRC offload"
+	default n
+	help
+	  Direct Data Placement (DDP) CRC32C offload for TCP enables ULP, such as
+	  NVMe-TCP/iSCSI, to request the NIC to calculate/verify the data digest
+	  of commands as they go through the NIC. Thus avoiding the costly
+	  per-byte overhead.
 
 endif   # if NET
 
net/ethtool/common.c +1 -0
index a2ff7a4a6bbf..4941670dd25d 100644
@@ -69,6 +69,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {
 	[NETIF_F_GRO_FRAGLIST_BIT] =	 "rx-gro-list",
 	[NETIF_F_HW_MACSEC_BIT] =	 "macsec-hw-offload",
 	[NETIF_F_HW_TCP_DDP_BIT] =	 "tcp-ddp-offload",
+	[NETIF_F_HW_TCP_DDP_CRC_BIT] =	 "tcp-ddp-crc-offload",
 };
 
 const char
net/ipv4/tcp_input.c +7 -0
index f7b3e37d2198..f8e1e8ebef2c 100644
@@ -5099,6 +5099,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 		memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
 #ifdef CONFIG_TLS_DEVICE
 		nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_TCP_DDP_CRC
+		nskb->ddp_crc = skb->ddp_crc;
 #endif
 		TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
 		if (list)
@@ -5132,6 +5135,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
 #ifdef CONFIG_TLS_DEVICE
 				if (skb->decrypted != nskb->decrypted)
 					goto end;
+#endif
+#ifdef CONFIG_TCP_DDP_CRC
+				if (skb->ddp_crc != nskb->ddp_crc)
+					goto end;
 #endif
 			}
 		}
net/ipv4/tcp_ipv4.c +3 -0
index ace48b2790ff..2693029e6ee7 100644
@@ -1791,6 +1791,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 	      TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_ECE | TCPHDR_CWR)) ||
 #ifdef CONFIG_TLS_DEVICE
 	    tail->decrypted != skb->decrypted ||
+#endif
+#ifdef CONFIG_TCP_DDP_CRC
+	    tail->ddp_crc != skb->ddp_crc ||
 #endif
 	    thtail->doff != th->doff ||
 	    memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)))
net/ipv4/tcp_offload.c +3 -0
index e09147ac9a99..39f5f0bcf181 100644
@@ -262,6 +262,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
 #ifdef CONFIG_TLS_DEVICE
 	flush |= p->decrypted ^ skb->decrypted;
 #endif
+#ifdef CONFIG_TCP_DDP_CRC
+	flush |= p->ddp_crc ^ skb->ddp_crc;
+#endif
 
 	if (flush || skb_gro_receive(p, skb)) {
 		mss = 1;
Signature · 1 line
2.24.1
maintainer Reviewed-by on v1 3/10
#7 lore raw

Re: [PATCH net-next RFC v1 03/10] net: Introduce crc offload for tcp ddp ulp

Quoted 2 lines

This commit itroduces support for CRC offload to direct data placement

introduces

Quoted 3 lines

ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.

From the API it not clear that the offload engine does crc32c, do you see this extended to other crc types in the future?

Other than this the patch looks good to me,

#8 lore raw

Re: [PATCH net-next RFC v1 03/10] net: Introduce crc offload for tcp ddp ulp

Quoted 7 lines · Sagi Grimberg
Quoted 1 line

This commit itroduces support for CRC offload to direct data placement

introduces

Quoted 3 lines

ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.

From the API it not clear that the offload engine does crc32c, do you
see this extended to other crc types in the future?

Yes, it is somewhat implicit, and it depends on the tcp ddp configuration. At the moment we only do nvme-tcp, and that has only CRC32C. If in the future, there would be other CRC variants, or data digest algorithms, then the code could be easily extended.

In general, any data digest over TCP can leverage this code and SKB member. Not only other CRC types can benefit from it, but even more complex data digest algorithms like SHA can use this. In essence this bit is similar to the TLS skb->decrypted bit. In TLS, skb->decrypted also indicates that the authentication has no error, exactly like ddp_crc indicates that there is no CRC32C error. The only reason we didn't use the same bit for both is that these two protocol offloads can be combined and that will benefit from two independent bits.

Quoted 3 lines

Other than this the patch looks good to me,

#9 lore raw

RFC PATCH 4/10 net/tls: expose get_netdev_for_sock

In reply to Boris Pismenny

get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.

Later patches will use this for nvme-tcp DDP and DDP CRC offloads.

---
 include/net/sock.h   | 17 +++++++++++++++++
 net/tls/tls_device.c | 20 ++------------------
 2 files changed, 19 insertions(+), 18 deletions(-)
include/net/sock.h +17 -0
index a5c6ae78df77..e0a92187f9ea 100644
@@ -2703,4 +2703,21 @@ void sock_set_sndtimeo(struct sock *sk, s64 secs);
 
 int sock_bind_add(struct sock *sk, struct sockaddr *addr, int addr_len);
 
+/* Assume that the socket is already connected */
+static inline struct net_device *get_netdev_for_sock(struct sock *sk, bool hold)
+{
+	struct dst_entry *dst = sk_dst_get(sk);
+	struct net_device *netdev = NULL;
+
+	if (likely(dst)) {
+		netdev = dst->dev;
+		if (hold)
+			dev_hold(netdev);
+	}
+
+	dst_release(dst);
+
+	return netdev;
+}
+
 #endif	/* _SOCK_H */
net/tls/tls_device.c +2 -18
index b74e2741f74f..4df0de65db04 100644
@@ -106,22 +106,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx)
 	spin_unlock_irqrestore(&tls_device_lock, flags);
 }
 
-/* We assume that the socket is already connected */
-static struct net_device *get_netdev_for_sock(struct sock *sk)
-{
-	struct dst_entry *dst = sk_dst_get(sk);
-	struct net_device *netdev = NULL;
-
-	if (likely(dst)) {
-		netdev = dst->dev;
-		dev_hold(netdev);
-	}
-
-	dst_release(dst);
-
-	return netdev;
-}
-
 static void destroy_record(struct tls_record_info *record)
 {
 	int i;
@@ -1086,7 +1070,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)
 	if (skb)
 		TCP_SKB_CB(skb)->eor = 1;
 
-	netdev = get_netdev_for_sock(sk);
+	netdev = get_netdev_for_sock(sk, true);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
 		rc = -EINVAL;
@@ -1162,7 +1146,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 	if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
 		return -EOPNOTSUPP;
 
-	netdev = get_netdev_for_sock(sk);
+	netdev = get_netdev_for_sock(sk, true);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
 		return -EINVAL;
Signature · 1 line
2.24.1
#11 lore raw

RFC PATCH 5/10 nvme-tcp: Add DDP offload control path

In reply to Boris Pismenny

This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.

Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.

Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via tcp_ddp_limits.

A follow-up patch introduces the data-path changes required for this offload.

---
 drivers/nvme/host/tcp.c  | 188 +++++++++++++++++++++++++++++++++++++++
 include/linux/nvme-tcp.h |   2 +
 2 files changed, 190 insertions(+)
drivers/nvme/host/tcp.c +188 -0
index 8f4f29f18b8c..06711ac095f2 100644
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_ALLOCATED	= 0,
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,
 };
 
 enum nvme_tcp_recv_state {
@@ -110,6 +111,8 @@ struct nvme_tcp_queue {
 	void (*state_change)(struct sock *);
 	void (*data_ready)(struct sock *);
 	void (*write_space)(struct sock *);
+
+	atomic64_t  resync_req;
 };
 
 struct nvme_tcp_ctrl {
@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {
 	struct delayed_work	connect_work;
 	struct nvme_tcp_request async_req;
 	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct net_device       *offloading_netdev;
 };
 
 static LIST_HEAD(nvme_tcp_ctrl_list);
@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 	return nvme_tcp_pdu_data_left(req) <= len;
 }
 
+#ifdef CONFIG_TCP_DDP
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops __read_mostly = {
+	.resync_request		= nvme_tcp_resync_request,
+};
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config;
+	int ret;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;
+	}
+
+	if (!(netdev->features & NETIF_F_HW_TCP_DDP)) {
+		dev_put(netdev);
+		return -EINVAL;
+	}
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add(netdev,
+						 queue->sock->sk,
+						 ddp_config);
+	if (!ret)
+		inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
+	else
+		dev_put(netdev);
+	return ret;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return;
+	}
+
+	netdev->tcp_ddp_ops->tcp_ddp_sk_del(netdev, queue->sock->sk);
+
+	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL;
+	dev_put(netdev); /* put the queue_init get_netdev_for_sock() */
+}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	int ret = 0;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;
+	}
+
+	if (netdev->features & NETIF_F_HW_TCP_DDP &&
+	    netdev->tcp_ddp_ops &&
+	    netdev->tcp_ddp_ops->tcp_ddp_limits)
+			ret = netdev->tcp_ddp_ops->tcp_ddp_limits(netdev, limits);
+	else
+			ret = -EOPNOTSUPP;
+
+	if (!ret) {
+		queue->ctrl->offloading_netdev = netdev;
+		pr_info("%s netdev %s offload limits: max_ddp_sgl_len %d\n",
+			__func__, netdev->name, limits->max_ddp_sgl_len);
+		queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
+		queue->ctrl->ctrl.max_hw_sectors =
+			limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
+	} else {
+		queue->ctrl->offloading_netdev = NULL;
+	}
+
+	dev_put(netdev);
+
+	return ret;
+}
+
+static
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	u64 resync_val;
+	u32 resync_seq;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return;
+	}
+
+	resync_val = atomic64_read(&queue->resync_req);
+	if ((resync_val & TCP_DDP_RESYNC_REQ) == 0)
+		return;
+
+	resync_seq = resync_val >> 32;
+	if (before(pdu_seq, resync_seq))
+		return;
+
+	if (atomic64_cmpxchg(&queue->resync_req, resync_val, (resync_val - 1)))
+		netdev->tcp_ddp_ops->tcp_ddp_resync(netdev, queue->sock->sk, pdu_seq);
+}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	struct nvme_tcp_queue *queue = sk->sk_user_data;
+
+	atomic64_set(&queue->resync_req,
+		     (((uint64_t)seq << 32) | flags));
+
+	return true;
+}
+
+#else
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	return false;
+}
+
+#endif
+
 static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
 		unsigned int dir)
 {
@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 	size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
 	int ret;
 
+	u64 pdu_seq = TCP_SKB_CB(skb)->seq + *offset - queue->pdu_offset;
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_resync_response(queue, pdu_seq);
+
 	ret = skb_copy_bits(skb, *offset,
 		&pdu[queue->pdu_offset], rcv_len);
 	if (unlikely(ret))
@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
 {
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
 	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
+	struct nvme_tcp_config config;
+	struct tcp_ddp_limits limits;
 	int ret, rcv_pdu_size;
 
 	queue->ctrl = ctrl;
@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
 #endif
 	write_unlock_bh(&queue->sock->sk->sk_callback_lock);
 
+	if (nvme_tcp_queue_id(queue) != 0) {
+		config.cfg.type		= TCP_DDP_NVME;
+		config.pfv		= NVME_TCP_PFV_1_0;
+		config.cpda		= 0;
+		config.dgst		= queue->hdr_digest ?
+						NVME_TCP_HDR_DIGEST_ENABLE : 0;
+		config.dgst		|= queue->data_digest ?
+						NVME_TCP_DATA_DIGEST_ENABLE : 0;
+		config.queue_size	= queue->queue_size;
+		config.queue_id		= nvme_tcp_queue_id(queue);
+		config.io_cpu		= queue->io_cpu;
+
+		ret = nvme_tcp_offload_socket(queue, &config);
+		if (!ret)
+			set_bit(NVME_TCP_Q_OFFLOADS, &queue->flags);
+	} else {
+		ret = nvme_tcp_offload_limits(queue, &limits);
+	}
+	/* offload is opportunistic - failure is non-critical */
+
 	return 0;
 
 err_init_connect:
@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
 	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
 	nvme_tcp_restore_sock_calls(queue);
 	cancel_work_sync(&queue->io_work);
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_unoffload_socket(queue);
 }
 
 static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
include/linux/nvme-tcp.h +2 -0
index 959e0bd9a913..65df64c34ecd 100644
@@ -8,6 +8,8 @@
 #define _LINUX_NVME_TCP_H
 
 #include <linux/nvme.h>
+#include <net/sock.h>
+#include <net/tcp_ddp.h>
 
 #define NVME_TCP_DISC_PORT	8009
 #define NVME_TCP_ADMIN_CCSZ	SZ_8K
Signature · 1 line
2.24.1
maintainer
#12 lore raw

Re: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path

Quoted 75 lines · Boris Pismenny

This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.

Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.

Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via tcp_ddp_limits.

A follow-up patch introduces the data-path changes required for this offload.

---
  drivers/nvme/host/tcp.c  | 188 +++++++++++++++++++++++++++++++++++++++
  include/linux/nvme-tcp.h |   2 +
  2 files changed, 190 insertions(+)
drivers/nvme/host/tcp.c +11 -0
index 8f4f29f18b8c..06711ac095f2 100644
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
  	NVME_TCP_Q_ALLOCATED	= 0,
  	NVME_TCP_Q_LIVE		= 1,
  	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,
  };
 
  enum nvme_tcp_recv_state {
@@ -110,6 +111,8 @@ struct nvme_tcp_queue {
  	void (*state_change)(struct sock *);
  	void (*data_ready)(struct sock *);
  	void (*write_space)(struct sock *);
+
+	atomic64_t  resync_req;
  };
 
  struct nvme_tcp_ctrl {
@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {
  	struct delayed_work	connect_work;
  	struct nvme_tcp_request async_req;
  	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct net_device       *offloading_netdev;
  };
 
  static LIST_HEAD(nvme_tcp_ctrl_list);
@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
  	return nvme_tcp_pdu_data_left(req) <= len;
  }
 
+#ifdef CONFIG_TCP_DDP
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops __read_mostly = {
+	.resync_request		= nvme_tcp_resync_request,
+};

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 246

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config;
+	int ret;
+
+	if (unlikely(!netdev)) {

Let's remove unlikely from non datapath routines, its slightly confusing.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 247

Quoted 1 line
drivers/nvme/host/tcp.c +1 -0
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

dev_info_ratelimited with queue->ctrl->ctrl.device ? Also, lets remove __func__. This usually is not very helpful.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 253

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+		return -EINVAL;
+	}
+
+	if (!(netdev->features & NETIF_F_HW_TCP_DDP)) {
+		dev_put(netdev);
+		return -EINVAL;

EINVAL or ENODEV?

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+	}
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add(netdev,
+						 queue->sock->sk,
+						 ddp_config);
+	if (!ret)
+		inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
+	else
+		dev_put(netdev);

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 272

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+	return ret;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

Same here.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 278

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+		return;
+	}
+
+	netdev->tcp_ddp_ops->tcp_ddp_sk_del(netdev, queue->sock->sk);
+
+	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL;

Just a general question, why is this needed?

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 291

Quoted 13 lines
drivers/nvme/host/tcp.c +13 -0
+	dev_put(netdev); /* put the queue_init get_netdev_for_sock() */
+}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	int ret = 0;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;

Same here

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 304

Quoted 13 lines
drivers/nvme/host/tcp.c +13 -0
+	}
+
+	if (netdev->features & NETIF_F_HW_TCP_DDP &&
+	    netdev->tcp_ddp_ops &&
+	    netdev->tcp_ddp_ops->tcp_ddp_limits)
+			ret = netdev->tcp_ddp_ops->tcp_ddp_limits(netdev, limits);
+	else
+			ret = -EOPNOTSUPP;
+
+	if (!ret) {
+		queue->ctrl->offloading_netdev = netdev;
+		pr_info("%s netdev %s offload limits: max_ddp_sgl_len %d\n",
+			__func__, netdev->name, limits->max_ddp_sgl_len);

dev_info, and given that it per-queue, please make it dev_dbg.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 309

Quoted 5 lines
drivers/nvme/host/tcp.c +5 -0
+		queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
+		queue->ctrl->ctrl.max_hw_sectors =
+			limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
+	} else {
+		queue->ctrl->offloading_netdev = NULL;

Maybe nullify in the controller setup intead?

Quoted 8 lines
drivers/nvme/host/tcp.c +8 -0
+	}
+
+	dev_put(netdev);
+
+	return ret;
+}
+
+static

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 327

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	u64 resync_val;
+	u32 resync_seq;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return;

What happens now, fallback to SW? Maybe dev_warn then.. will the SW keep seeing these responses after one failed?

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 336

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+	}
+
+	resync_val = atomic64_read(&queue->resync_req);
+	if ((resync_val & TCP_DDP_RESYNC_REQ) == 0)
+		return;
+
+	resync_seq = resync_val >> 32;
+	if (before(pdu_seq, resync_seq))
+		return;

I think it will be better to pass the skb to this func and keep the pdu_seq contained locally.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 339

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+
+	if (atomic64_cmpxchg(&queue->resync_req, resync_val, (resync_val - 1)))
+		netdev->tcp_ddp_ops->tcp_ddp_resync(netdev, queue->sock->sk, pdu_seq);

A small comment on this manipulation may help the reader.

Quoted 48 lines
drivers/nvme/host/tcp.c +44 -0
+}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	struct nvme_tcp_queue *queue = sk->sk_user_data;
+
+	atomic64_set(&queue->resync_req,
+		     (((uint64_t)seq << 32) | flags));
+
+	return true;
+}
+
+#else
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	return false;
+}
+
+#endif
+
  static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
  		unsigned int dir)
  {

drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 792

Quoted 7 lines
drivers/nvme/host/tcp.c +4 -0
@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  	size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
  	int ret;
 
+	u64 pdu_seq = TCP_SKB_CB(skb)->seq + *offset - queue->pdu_offset;
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_resync_response(queue, pdu_seq);

Here, just pass in (queue, skb)

Quoted 14 lines
drivers/nvme/host/tcp.c +3 -0
+
  	ret = skb_copy_bits(skb, *offset,
  		&pdu[queue->pdu_offset], rcv_len);
  	if (unlikely(ret))
@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
  {
  	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
  	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
+	struct nvme_tcp_config config;
+	struct tcp_ddp_limits limits;
  	int ret, rcv_pdu_size;
 
  	queue->ctrl = ctrl;

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1655

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
  #endif
  	write_unlock_bh(&queue->sock->sk->sk_callback_lock);
 
+	if (nvme_tcp_queue_id(queue) != 0) {
	if (!nvme_tcp_admin_queue(queue)) {

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1665

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+		config.cfg.type		= TCP_DDP_NVME;
+		config.pfv		= NVME_TCP_PFV_1_0;
+		config.cpda		= 0;
+		config.dgst		= queue->hdr_digest ?
+						NVME_TCP_HDR_DIGEST_ENABLE : 0;
+		config.dgst		|= queue->data_digest ?
+						NVME_TCP_DATA_DIGEST_ENABLE : 0;
+		config.queue_size	= queue->queue_size;
+		config.queue_id		= nvme_tcp_queue_id(queue);
+		config.io_cpu		= queue->io_cpu;

Can the config initialization move to nvme_tcp_offload_socket?

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1672

Quoted 7 lines
drivers/nvme/host/tcp.c +7 -0
+
+		ret = nvme_tcp_offload_socket(queue, &config);
+		if (!ret)
+			set_bit(NVME_TCP_Q_OFFLOADS, &queue->flags);
+	} else {
+		ret = nvme_tcp_offload_limits(queue, &limits);
+	}

I'm thinking that instead of this conditional, we want to place nvme nvme_tcp_alloc_admin_queue in nvme_tcp_alloc_admin_queue, and also move nvme_tcp_alloc_admin_queue to __nvme_tcp_alloc_io_queues loop.

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1673

Quoted 1 line
drivers/nvme/host/tcp.c +1 -0
+	/* offload is opportunistic - failure is non-critical */

Than make it void...

drivers/nvme/host/tcp.c static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) · line 1709

Quoted 11 lines
drivers/nvme/host/tcp.c +4 -0
+
  	return 0;
 
  err_init_connect:
@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
  	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
  	nvme_tcp_restore_sock_calls(queue);
  	cancel_work_sync(&queue->io_work);
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_unoffload_socket(queue);

Why not in nvme_tcp_free_queue, symmetric to the alloc?

include/linux/nvme-tcp.h line 12

Quoted 13 lines
drivers/nvme/host/tcp.c +0 -0
  }
 
  static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
include/linux/nvme-tcp.h +2 -0
index 959e0bd9a913..65df64c34ecd 100644
@@ -8,6 +8,8 @@
  #define _LINUX_NVME_TCP_H
 
  #include <linux/nvme.h>
+#include <net/sock.h>
+#include <net/tcp_ddp.h>

Why is this needed? I think we want to place this in tcp.c no?

#13 lore raw

Re: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path

Quoted 93 lines · Sagi Grimberg
Quoted 75 lines · Boris Pismenny

This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.

Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.

Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via tcp_ddp_limits.

A follow-up patch introduces the data-path changes required for this offload.

---
  drivers/nvme/host/tcp.c  | 188 +++++++++++++++++++++++++++++++++++++++
  include/linux/nvme-tcp.h |   2 +
  2 files changed, 190 insertions(+)
drivers/nvme/host/tcp.c +11 -0
index 8f4f29f18b8c..06711ac095f2 100644
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
  	NVME_TCP_Q_ALLOCATED	= 0,
  	NVME_TCP_Q_LIVE		= 1,
  	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,
  };
 
  enum nvme_tcp_recv_state {
@@ -110,6 +111,8 @@ struct nvme_tcp_queue {
  	void (*state_change)(struct sock *);
  	void (*data_ready)(struct sock *);
  	void (*write_space)(struct sock *);
+
+	atomic64_t  resync_req;
  };
 
  struct nvme_tcp_ctrl {
@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {
  	struct delayed_work	connect_work;
  	struct nvme_tcp_request async_req;
  	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct net_device       *offloading_netdev;
  };
 
  static LIST_HEAD(nvme_tcp_ctrl_list);
@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
  	return nvme_tcp_pdu_data_left(req) <= len;
  }
 
+#ifdef CONFIG_TCP_DDP
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops __read_mostly = {
+	.resync_request		= nvme_tcp_resync_request,
+};

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 246

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config;
+	int ret;
+
+	if (unlikely(!netdev)) {

Let's remove unlikely from non datapath routines, its slightly confusing.

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 247

Quoted 1 line
drivers/nvme/host/tcp.c +1 -0
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

dev_info_ratelimited with queue->ctrl->ctrl.device ? Also, lets remove __func__. This usually is not very helpful.

Quoted 7 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 253

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+		return -EINVAL;
+	}
+
+	if (!(netdev->features & NETIF_F_HW_TCP_DDP)) {
+		dev_put(netdev);
+		return -EINVAL;

EINVAL or ENODEV?

ENODEV seems more appropriate, we'll use it. Thanks

Quoted 21 lines
Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+	}
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add(netdev,
+						 queue->sock->sk,
+						 ddp_config);
+	if (!ret)
+		inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
+	else
+		dev_put(netdev);

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 272

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+	return ret;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

Same here.

Quoted 7 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 278

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+		return;
+	}
+
+	netdev->tcp_ddp_ops->tcp_ddp_sk_del(netdev, queue->sock->sk);
+
+	inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL;

Just a general question, why is this needed?

This assignment is symmetric with the nvme_tcp_offload_socket assignment. The idea was to ensure that the functions established during offload cannot be used from this moment.

Quoted 30 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 291

Quoted 13 lines
drivers/nvme/host/tcp.c +13 -0
+	dev_put(netdev); /* put the queue_init get_netdev_for_sock() */
+}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true);
+	int ret = 0;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;

Same here

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 304

Quoted 13 lines
drivers/nvme/host/tcp.c +13 -0
+	}
+
+	if (netdev->features & NETIF_F_HW_TCP_DDP &&
+	    netdev->tcp_ddp_ops &&
+	    netdev->tcp_ddp_ops->tcp_ddp_limits)
+			ret = netdev->tcp_ddp_ops->tcp_ddp_limits(netdev, limits);
+	else
+			ret = -EOPNOTSUPP;
+
+	if (!ret) {
+		queue->ctrl->offloading_netdev = netdev;
+		pr_info("%s netdev %s offload limits: max_ddp_sgl_len %d\n",
+			__func__, netdev->name, limits->max_ddp_sgl_len);

dev_info, and given that it per-queue, please make it dev_dbg.

Quoted 6 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 309

Quoted 5 lines
drivers/nvme/host/tcp.c +5 -0
+		queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
+		queue->ctrl->ctrl.max_hw_sectors =
+			limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
+	} else {
+		queue->ctrl->offloading_netdev = NULL;

Maybe nullify in the controller setup intead?

It is already set to zero after allocation (i.e., kzalloc). The goal here is to ensure it is zero in case there is a reset and it was non zero due to offload being used in the past.

Quoted 12 lines
Quoted 12 lines
drivers/nvme/host/tcp.c +12 -0
+	}
+
+	dev_put(netdev);
+
+	return ret;
+}
+
+static
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
Quoted 8 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 327

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+	u64 resync_val;
+	u32 resync_seq;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return;

What happens now, fallback to SW? Maybe dev_warn then.. will the SW keep seeing these responses after one failed?

As long as there is no resync response the system falls back to software, and this message is emitted. I'll move it to below to display it only when it is relevant and not every time this function is called.

Quoted 11 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 336

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+	}
+
+	resync_val = atomic64_read(&queue->resync_req);
+	if ((resync_val & TCP_DDP_RESYNC_REQ) == 0)
+		return;
+
+	resync_seq = resync_val >> 32;
+	if (before(pdu_seq, resync_seq))
+		return;

I think it will be better to pass the skb to this func and keep the pdu_seq contained locally.

This requires passing the offset to obtain the sequence:
u64 pdu_seq = TCP_SKB_CB(skb)->seq + *offset - queue->pdu_offset;

It makes the interface a bit ugly, IMO.

Quoted 56 lines

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 339

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+
+	if (atomic64_cmpxchg(&queue->resync_req, resync_val, (resync_val - 1)))
+		netdev->tcp_ddp_ops->tcp_ddp_resync(netdev, queue->sock->sk, pdu_seq);

A small comment on this manipulation may help the reader.

Quoted 51 lines
drivers/nvme/host/tcp.c +44 -0
+}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	struct nvme_tcp_queue *queue = sk->sk_user_data;
+
+	atomic64_set(&queue->resync_req,
+		     (((uint64_t)seq << 32) | flags));
+
+	return true;
+}
+
+#else
+
+static
+int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
+			    struct nvme_tcp_config *config)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{}
+
+static
+int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
+			    struct tcp_ddp_limits *limits)
+{
+	return -EINVAL;
+}
+
+static
+void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+			      unsigned int pdu_seq)
+{}
+
+bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+	return false;
+}
+
+#endif
+
  static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
  		unsigned int dir)
  {
@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  	size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
  	int ret;
 
Quoted 5 lines

drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 792

Quoted 4 lines
drivers/nvme/host/tcp.c +4 -0
+	u64 pdu_seq = TCP_SKB_CB(skb)->seq + *offset - queue->pdu_offset;
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_resync_response(queue, pdu_seq);

Here, just pass in (queue, skb)

Do you mean (queue, skb, offset)?
We need the offset for the pdu_seq calculation above.

Quoted 20 lines
Quoted 14 lines
drivers/nvme/host/tcp.c +3 -0
+
  	ret = skb_copy_bits(skb, *offset,
  		&pdu[queue->pdu_offset], rcv_len);
  	if (unlikely(ret))
@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
  {
  	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
  	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
+	struct nvme_tcp_config config;
+	struct tcp_ddp_limits limits;
  	int ret, rcv_pdu_size;
 
  	queue->ctrl = ctrl;

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1655

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,
  #endif
  	write_unlock_bh(&queue->sock->sk->sk_callback_lock);
 
+	if (nvme_tcp_queue_id(queue) != 0) {
	if (!nvme_tcp_admin_queue(queue)) {
Quoted 11 lines

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1665

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+		config.cfg.type		= TCP_DDP_NVME;
+		config.pfv		= NVME_TCP_PFV_1_0;
+		config.cpda		= 0;
+		config.dgst		= queue->hdr_digest ?
+						NVME_TCP_HDR_DIGEST_ENABLE : 0;
+		config.dgst		|= queue->data_digest ?
+						NVME_TCP_DATA_DIGEST_ENABLE : 0;
+		config.queue_size	= queue->queue_size;
+		config.queue_id		= nvme_tcp_queue_id(queue);
+		config.io_cpu		= queue->io_cpu;

Can the config initialization move to nvme_tcp_offload_socket?

Definitely. The original idea of placing it here was that the nvme-tcp handshake may influence the parameters. But, at this moment, I realize that it is orthogonal.

Quoted 11 lines

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1672

Quoted 7 lines
drivers/nvme/host/tcp.c +7 -0
+
+		ret = nvme_tcp_offload_socket(queue, &config);
+		if (!ret)
+			set_bit(NVME_TCP_Q_OFFLOADS, &queue->flags);
+	} else {
+		ret = nvme_tcp_offload_limits(queue, &limits);
+	}

I'm thinking that instead of this conditional, we want to place nvme nvme_tcp_alloc_admin_queue in nvme_tcp_alloc_admin_queue, and also move nvme_tcp_alloc_admin_queue to __nvme_tcp_alloc_io_queues loop.

We need to do it only after the socket is connected to obtain the 5-tuple and the appropriate netdev, and preferably after the protocol handshake, as there is nothing to offload there.

Quoted 6 lines

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1673

Quoted 1 line
drivers/nvme/host/tcp.c +1 -0
+	/* offload is opportunistic - failure is non-critical */

Than make it void...

Quoted 3 lines
drivers/nvme/host/tcp.c +1 -0
+
  	return 0;
 
Quoted 9 lines

drivers/nvme/host/tcp.c static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) · line 1709

Quoted 8 lines
drivers/nvme/host/tcp.c +3 -0
  err_init_connect:
@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
  	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
  	nvme_tcp_restore_sock_calls(queue);
  	cancel_work_sync(&queue->io_work);
+
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_unoffload_socket(queue);

Why not in nvme_tcp_free_queue, symmetric to the alloc?

Sure. I've tried to keep it close to the socket disconnect, which isn't symmetrical for some reason, which suggests that these are interchangable?!

Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +0 -0
  }
 
  static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
include/linux/nvme-tcp.h +0 -0
index 959e0bd9a913..65df64c34ecd 100644
@@ -8,6 +8,8 @@
  #define _LINUX_NVME_TCP_H
 
Quoted 4 lines

include/linux/nvme-tcp.h line 12

Quoted 3 lines
include/linux/nvme-tcp.h +2 -0
  #include <linux/nvme.h>
+#include <net/sock.h>
+#include <net/tcp_ddp.h>

Why is this needed? I think we want to place this in tcp.c no?

Not needed. It's probably leftover from previous iterations on the code. Removed for the next iteration of the patchset.

A message that is not in the archive (sent off-list or not yet received): PH0PR18MB3845430DDF572E0DD4832D06CCED0@PH0PR18MB3845.namprd18.prod.outlook.com

#15 lore raw

RE: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path

Quoted 38 lines · Sagi Grimberg
Quoted 10 lines · Boris Pismenny

This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the tcp_ddp_sk_add/del NDOs.

Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload 2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to

confirm

Quoted 16 lines

it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW, and resume offload when all is successful.

Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via tcp_ddp_limits.

A follow-up patch introduces the data-path changes required for this offload.

Quoted 6 lines
---
  drivers/nvme/host/tcp.c  | 188

+++++++++++++++++++++++++++++++++++++++

Quoted 3 lines
  include/linux/nvme-tcp.h |   2 +
  2 files changed, 190 insertions(+)
Quoted 9 lines

drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65

Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0 -0

8f4f29f18b8c..06711ac095f2 100644

drivers/nvme/host/tcp.c +1 -0
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
  	NVME_TCP_Q_ALLOCATED	= 0,
  	NVME_TCP_Q_LIVE		= 1,
  	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,

Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a TCP_OFFLOAD transport type would be added, we are concerned that perhaps the generic term "offload" for both the transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g. NVME_TCP_Q_DDP or NVME_TCP_Q_DIRECT? Also, no need to quote the entire patch. Just a few lines above your response like I did here.

maintainer
#16 lore raw

Re: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path

Quoted 12 lines
Quoted 10 lines

drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65

Quoted 10 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0 -0

8f4f29f18b8c..06711ac095f2 100644

drivers/nvme/host/tcp.c +1 -0
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
   	NVME_TCP_Q_ALLOCATED	= 0,
   	NVME_TCP_Q_LIVE		= 1,
   	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,

Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a TCP_OFFLOAD transport type would be added,

We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.

Quoted 1 line

we are concerned that perhaps the generic term "offload" for both the transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g. NVME_TCP_Q_DDP or NVME_TCP_Q_DIRECT?

We can call this NVME_TCP_Q_DDP, no issues with that.

#17 lore raw

RFC PATCH 5/10 nvme-tcp: Add DDP offload control path

Quoted 14 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0 -0

8f4f29f18b8c..06711ac095f2 100644

drivers/nvme/host/tcp.c +0 -0
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
Quoted 8 lines
Quoted 4 lines

drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
   	NVME_TCP_Q_ALLOCATED	= 0,
   	NVME_TCP_Q_LIVE		= 1,
   	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,

Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a TCP_OFFLOAD transport type would be added,

Quoted 4 lines

We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.

We plan to do this soon.

Quoted 8 lines
Quoted 1 line

we are concerned that perhaps the generic term "offload" for both the

transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g. NVME_TCP_Q_DDP or NVME_TCP_Q_DIRECT?

We can call this NVME_TCP_Q_DDP, no issues with that.

Great. Thanks.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#18 lore raw

RE: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path

Quoted 15 lines · Sagi Grimberg
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/tcp.c +0 -0

index
8f4f29f18b8c..06711ac095f2 100644

drivers/nvme/host/tcp.c +0 -0
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {
Quoted 8 lines
Quoted 4 lines

drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
   	NVME_TCP_Q_ALLOCATED	= 0,
   	NVME_TCP_Q_LIVE		= 1,
   	NVME_TCP_Q_POLLING	= 2,
+	NVME_TCP_Q_OFFLOADS     = 3,

Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a TCP_OFFLOAD transport type would be added,

Quoted 4 lines

We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.

We plan to do this soon.

Quoted 9 lines
Quoted 2 lines

we are concerned that perhaps the generic term "offload" for both the

transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g. NVME_TCP_Q_DDP or NVME_TCP_Q_DIRECT?

We can call this NVME_TCP_Q_DDP, no issues with that.

Great. Thanks.

#19 lore raw

RFC PATCH 6/10 nvme-tcp: Add DDP data-path

In reply to Boris Pismenny

Introduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id in the PDU.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.

The data-path interface contains two routines: tcp_ddp_setup/teardown. The setup provides the mapping from command_id to the request buffers, while the teardown removes this mapping.

For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the generic mq_blk completion.

---
 drivers/nvme/host/tcp.c | 121 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 117 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +117 -4
index 06711ac095f2..7bd97f856677 100644
@@ -56,6 +56,11 @@ struct nvme_tcp_request {
 	size_t			offset;
 	size_t			data_sent;
 	enum nvme_tcp_send_state state;
+
+	bool			offloaded;
+	struct tcp_ddp_io	ddp;
+	__le16			status;
+	union nvme_result	result;
 };
 
 enum nvme_tcp_queue_flags {
@@ -231,10 +236,76 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 #ifdef CONFIG_TCP_DDP
 
 bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
 const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops __read_mostly = {
+
 	.resync_request		= nvme_tcp_resync_request,
+	.ddp_teardown_done	= nvme_tcp_ddp_teardown_done,
 };
 
+static
+int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+			  uint16_t command_id,
+			  struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;
+	}
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_teardown(netdev, queue->sock->sk,
+						    &req->ddp, rq);
+	sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+	req->offloaded = false;
+	return ret;
+}
+
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{
+	struct request *rq = ddp_ctx;
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(req->status << 1), req->result))
+		nvme_complete_rq(rq);
+}
+
+static
+int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+		       uint16_t command_id,
+		       struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	req->offloaded = false;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;
+	}
+
+	req->ddp.command_id = command_id;
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	ret = sg_alloc_table_chained(&req->ddp.sg_table,
+		blk_rq_nr_phys_segments(rq), req->ddp.sg_table.sgl,
+		SG_CHUNK_SIZE);
+	if (ret)
+		return -ENOMEM;
+	req->ddp.nents = blk_rq_map_sg(rq->q, rq, req->ddp.sg_table.sgl);
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_setup(netdev,
+						 queue->sock->sk,
+						 &req->ddp);
+	if (!ret)
+		req->offloaded = true;
+	return ret;
+}
+
 static
 int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
 			    struct nvme_tcp_config *config)
@@ -351,6 +422,25 @@ bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
 
 #else
 
+static
+int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+		       uint16_t command_id,
+		       struct request *rq)
+{
+	return -EINVAL;
+}
+
+static
+int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+			  uint16_t command_id,
+			  struct request *rq)
+{
+	return -EINVAL;
+}
+
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{}
+
 static
 int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
 			    struct nvme_tcp_config *config)
@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
 static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 		struct nvme_completion *cqe)
 {
+	struct nvme_tcp_request *req;
 	struct request *rq;
 
 	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);
@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 		return -EINVAL;
 	}
 
-	if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
-		nvme_complete_rq(rq);
+	req = blk_mq_rq_to_pdu(rq);
+	if (req->offloaded) {
+		req->status = cqe->status;
+		req->result = cqe->result;
+		nvme_tcp_teardown_ddp(queue, cqe->command_id, rq);
+	} else {
+		if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
+			nvme_complete_rq(rq);
+	}
 	queue->nr_cqe++;
 
 	return 0;
@@ -836,9 +934,18 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 static inline void nvme_tcp_end_request(struct request *rq, u16 status)
 {
 	union nvme_result res = {};
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_queue *queue = req->queue;
+	struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
 
-	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
-		nvme_complete_rq(rq);
+	if (req->offloaded) {
+		req->status = cpu_to_le16(status << 1);
+		req->result = res;
+		nvme_tcp_teardown_ddp(queue, pdu->command_id, rq);
+	} else {
+		if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
+			nvme_complete_rq(rq);
+	}
 }
 
 static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	bool inline_data = nvme_tcp_has_inline_data(req);
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) + hdgst - req->offset;
+	struct request *rq = blk_mq_rq_from_pdu(req);
 	int flags = MSG_DONTWAIT;
 	int ret;
 
@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	else
 		flags |= MSG_EOR;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+	    blk_rq_nr_phys_segments(rq) && rq_data_dir(rq) == READ)
+		nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq);
+
 	if (queue->hdr_digest && !req->offset)
 		nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu));
 
@@ -2448,6 +2560,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,
 	req->data_len = blk_rq_nr_phys_segments(rq) ?
 				blk_rq_payload_bytes(rq) : 0;
 	req->curr_bio = rq->bio;
+	req->offloaded = false;
 
 	if (rq_data_dir(rq) == WRITE &&
 	    req->data_len <= nvme_tcp_inline_data_size(queue))
Signature · 1 line
2.24.1
maintainer
#20 lore raw

Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path

Quoted 10 lines
  bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
drivers/nvme/host/tcp.c +4 -0
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
  const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops __read_mostly = {
+
  	.resync_request		= nvme_tcp_resync_request,
+	.ddp_teardown_done	= nvme_tcp_ddp_teardown_done,
  };
 
+static

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 256

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+			  uint16_t command_id,
+			  struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

dev_info_ratelimited

Quoted 32 lines
drivers/nvme/host/tcp.c +32 -0
+		return -EINVAL;
+	}
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_teardown(netdev, queue->sock->sk,
+						    &req->ddp, rq);
+	sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+	req->offloaded = false;
+	return ret;
+}
+
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{
+	struct request *rq = ddp_ctx;
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(req->status << 1), req->result))
+		nvme_complete_rq(rq);
+}
+
+static
+int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+		       uint16_t command_id,
+		       struct request *rq)
+{
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct net_device *netdev = queue->ctrl->offloading_netdev;
+	int ret;
+
+	req->offloaded = false;
+
+	if (unlikely(!netdev)) {
+		pr_info_ratelimited("%s: netdev not found\n", __func__);

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 298

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+		return -EINVAL;
+	}
+
+	req->ddp.command_id = command_id;
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	ret = sg_alloc_table_chained(&req->ddp.sg_table,
+		blk_rq_nr_phys_segments(rq), req->ddp.sg_table.sgl,
+		SG_CHUNK_SIZE);
+	if (ret)
+		return -ENOMEM;

newline here

Quoted 95 lines
drivers/nvme/host/tcp.c +51 -4
+	req->ddp.nents = blk_rq_map_sg(rq->q, rq, req->ddp.sg_table.sgl);
+
+	ret = netdev->tcp_ddp_ops->tcp_ddp_setup(netdev,
+						 queue->sock->sk,
+						 &req->ddp);
+	if (!ret)
+		req->offloaded = true;
+	return ret;
+}
+
  static
  int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
  			    struct nvme_tcp_config *config)
@@ -351,6 +422,25 @@ bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
 
  #else
 
+static
+int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+		       uint16_t command_id,
+		       struct request *rq)
+{
+	return -EINVAL;
+}
+
+static
+int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+			  uint16_t command_id,
+			  struct request *rq)
+{
+	return -EINVAL;
+}
+
+void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{}
+
  static
  int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue,
  			    struct nvme_tcp_config *config)
@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
  static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
  		struct nvme_completion *cqe)
  {
+	struct nvme_tcp_request *req;
  	struct request *rq;
 
  	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);
@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
  		return -EINVAL;
  	}
 
-	if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
-		nvme_complete_rq(rq);
+	req = blk_mq_rq_to_pdu(rq);
+	if (req->offloaded) {
+		req->status = cqe->status;
+		req->result = cqe->result;
+		nvme_tcp_teardown_ddp(queue, cqe->command_id, rq);
+	} else {
+		if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
+			nvme_complete_rq(rq);
+	}
  	queue->nr_cqe++;
 
  	return 0;
@@ -836,9 +934,18 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  static inline void nvme_tcp_end_request(struct request *rq, u16 status)
  {
  	union nvme_result res = {};
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_queue *queue = req->queue;
+	struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
 
-	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
-		nvme_complete_rq(rq);
+	if (req->offloaded) {
+		req->status = cpu_to_le16(status << 1);
+		req->result = res;
+		nvme_tcp_teardown_ddp(queue, pdu->command_id, rq);
+	} else {
+		if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
+			nvme_complete_rq(rq);
+	}
  }
 
  static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
  	bool inline_data = nvme_tcp_has_inline_data(req);
  	u8 hdgst = nvme_tcp_hdgst_len(queue);
  	int len = sizeof(*pdu) + hdgst - req->offset;
+	struct request *rq = blk_mq_rq_from_pdu(req);
  	int flags = MSG_DONTWAIT;
  	int ret;
 

drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236

Quoted 6 lines
drivers/nvme/host/tcp.c +3 -0
@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
  	else
  		flags |= MSG_EOR;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+	    blk_rq_nr_phys_segments(rq) && rq_data_dir(rq) == READ)
+		nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq);

I'd assume that this is something we want to setup in nvme_tcp_setup_cmd_pdu. Why do it here?

maintainer
#21 lore raw

Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path

Quoted 19 lines
Quoted 19 lines

  static
  int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue, struct nvme_tcp_config *config)

drivers/nvme/host/tcp.c +2 -2
@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct 
 nvme_ctrl *ctrl)
   static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
           struct nvme_completion *cqe)
   {
+    struct nvme_tcp_request *req;
       struct request *rq;
       rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);
@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct 
 nvme_tcp_queue *queue,
           return -EINVAL;
       }
-    if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
-        nvme_complete_rq(rq);
+    req = blk_mq_rq_to_pdu(rq);
Quoted 8 lines

drivers/nvme/host/tcp.c static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · line 743

Quoted 8 lines
drivers/nvme/host/tcp.c +8 -0
+    if (req->offloaded) {
+        req->status = cqe->status;
+        req->result = cqe->result;
+        nvme_tcp_teardown_ddp(queue, cqe->command_id, rq);
+    } else {
+        if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
+            nvme_complete_rq(rq);
+    }

Oh forgot to ask,

We have places in the driver that we may complete (cancel) one or more requests from the error recovery or timeout flow. We first prevent future incoming RX on the socket such that we can safely cancel requests. This may break with the deferred completion in ddp_teardown_done.

If I have a request that is waiting for ddp_teardown_done do I have a way to tell the HW to never call ddp_teardown_done on a specific socket?

If so the place to is in nvme_tcp_stop_queue.

#22 lore raw

Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path

Quoted 34 lines · Sagi Grimberg
Quoted 18 lines
Quoted 18 lines

  static
  int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue, struct nvme_tcp_config *config)

drivers/nvme/host/tcp.c +2 -2
@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct 
 nvme_ctrl *ctrl)
   static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
           struct nvme_completion *cqe)
   {
+    struct nvme_tcp_request *req;
       struct request *rq;
       rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);
@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct 
 nvme_tcp_queue *queue,
           return -EINVAL;
       }
-    if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
-        nvme_complete_rq(rq);
+    req = blk_mq_rq_to_pdu(rq);
Quoted 8 lines

drivers/nvme/host/tcp.c static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · line 743

Quoted 8 lines
drivers/nvme/host/tcp.c +8 -0
+    if (req->offloaded) {
+        req->status = cqe->status;
+        req->result = cqe->result;
+        nvme_tcp_teardown_ddp(queue, cqe->command_id, rq);
+    } else {
+        if (!nvme_try_complete_req(rq, cqe->status, cqe->result))
+            nvme_complete_rq(rq);
+    }

Oh forgot to ask,

We have places in the driver that we may complete (cancel) one or more requests from the error recovery or timeout flow. We first prevent future incoming RX on the socket such that we can safely cancel requests. This may break with the deferred completion in ddp_teardown_done.

Quoted 5 lines

If I have a request that is waiting for ddp_teardown_done do I have a way to tell the HW to never call ddp_teardown_done on a specific socket?

If so the place to is in nvme_tcp_stop_queue.

Interesting and indeed, it is a problem that we haven't considered.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#23 lore raw

Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path

In reply to Sagi Grimberg
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines

static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,

drivers/nvme/host/tcp.c +1 -0
@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
  	bool inline_data = nvme_tcp_has_inline_data(req);
  	u8 hdgst = nvme_tcp_hdgst_len(queue);
  	int len = sizeof(*pdu) + hdgst - req->offset;
+	struct request *rq = blk_mq_rq_from_pdu(req);
  	int flags = MSG_DONTWAIT;
  	int ret;
 
@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
  	else
  		flags |= MSG_EOR;
 
Quoted 5 lines

drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+	    blk_rq_nr_phys_segments(rq) && rq_data_dir(rq) == READ)
+		nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq);

I'd assume that this is something we want to setup in nvme_tcp_setup_cmd_pdu. Why do it here?

Our goal in placing it here is to keep both setup and teardown in the same thread. This enables drivers to avoid locking for per-queue operations.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#24 lore raw

Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,

drivers/nvme/host/tcp.c +1 -0
@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
   	bool inline_data = nvme_tcp_has_inline_data(req);
   	u8 hdgst = nvme_tcp_hdgst_len(queue);
   	int len = sizeof(*pdu) + hdgst - req->offset;
+	struct request *rq = blk_mq_rq_from_pdu(req);
   	int flags = MSG_DONTWAIT;
   	int ret;
 
@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
   	else
   		flags |= MSG_EOR;
 
Quoted 7 lines
Quoted 5 lines

drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+	    blk_rq_nr_phys_segments(rq) && rq_data_dir(rq) == READ)
+		nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq);

I'd assume that this is something we want to setup in nvme_tcp_setup_cmd_pdu. Why do it here?

Our goal in placing it here is to keep both setup and teardown in the same thread. This enables drivers to avoid locking for per-queue operations.

I also think that it is cleaner when setting up the PDU. Do note that if queues match 1x1 with cpu cores then any synchronization is pretty lightweight, and if not, we have other synchronizations anyways...

#25 lore raw

RFC PATCH 7/10 nvme-tcp : Recalculate crc in the end of the capsule

In reply to Boris Pismenny

From: Yoray Zack <[email protected]>

crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.

This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.

---
 drivers/nvme/host/tcp.c | 66 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 58 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +58 -8
index 7bd97f856677..9a620d1dacb4 100644
@@ -94,6 +94,7 @@ struct nvme_tcp_queue {
 	size_t			data_remaining;
 	size_t			ddgst_remaining;
 	unsigned int		nr_cqe;
+	bool			crc_valid;
 
 	/* send state */
 	struct nvme_tcp_request *request;
@@ -233,6 +234,41 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
 	return nvme_tcp_pdu_data_left(req) <= len;
 }
 
+static inline bool nvme_tcp_device_ddgst_ok(struct nvme_tcp_queue *queue)
+{
+	return queue->crc_valid;
+}
+
+static inline void nvme_tcp_device_ddgst_update(struct nvme_tcp_queue *queue,
+						struct sk_buff *skb)
+{
+	if (queue->crc_valid)
+#ifdef CONFIG_TCP_DDP_CRC
+		queue->crc_valid = skb->ddp_crc;
+#else
+		queue->crc_valid = false;
+#endif
+}
+
+static void nvme_tcp_crc_recalculate(struct nvme_tcp_queue *queue,
+				     struct nvme_tcp_data_pdu *pdu)
+{
+	struct nvme_tcp_request *req;
+	struct request *rq;
+
+	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+	if (!rq)
+		return;
+	req = blk_mq_rq_to_pdu(rq);
+	crypto_ahash_init(queue->rcv_hash);
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	/* req->ddp.sg_table is allocated and filled in nvme_tcp_setup_ddp */
+	ahash_request_set_crypt(queue->rcv_hash, req->ddp.sg_table.sgl, NULL,
+				le32_to_cpu(pdu->data_length));
+	crypto_ahash_update(queue->rcv_hash);
+}
+
+
 #ifdef CONFIG_TCP_DDP
 
 bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
@@ -706,6 +742,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
 	queue->pdu_offset = 0;
 	queue->data_remaining = -1;
 	queue->ddgst_remaining = 0;
+	queue->crc_valid = true;
 }
 
 static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
@@ -955,6 +992,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 	struct nvme_tcp_request *req;
 	struct request *rq;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_device_ddgst_update(queue, skb);
 	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 		recv_len = min_t(size_t, recv_len,
 				iov_iter_count(&req->iter));
 
-		if (queue->data_digest)
+		if (queue->data_digest && !test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
 			ret = skb_copy_and_hash_datagram_iter(skb, *offset,
 				&req->iter, recv_len, queue->rcv_hash);
 		else
@@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
 	if (!queue->data_remaining) {
 		if (queue->data_digest) {
-			nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
 			queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
 		} else {
 			if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
 	char *ddgst = (char *)&queue->recv_ddgst;
 	size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining);
 	off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
+	bool ddgst_offload_fail;
 	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_device_ddgst_update(queue, skb);
 	ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len);
 	if (unlikely(ret))
 		return ret;
@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
 	if (queue->ddgst_remaining)
 		return 0;
 
-	if (queue->recv_ddgst != queue->exp_ddgst) {
-		dev_err(queue->ctrl->ctrl.device,
-			"data digest error: recv %#x expected %#x\n",
-			le32_to_cpu(queue->recv_ddgst),
-			le32_to_cpu(queue->exp_ddgst));
-		return -EIO;
+	ddgst_offload_fail = !nvme_tcp_device_ddgst_ok(queue);
+	if (!test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) ||
+	    ddgst_offload_fail) {
+		if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+		    ddgst_offload_fail)
+			nvme_tcp_crc_recalculate(queue, pdu);
+
+		nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
+		if (queue->recv_ddgst != queue->exp_ddgst) {
+			dev_err(queue->ctrl->ctrl.device,
+				"data digest error: recv %#x expected %#x\n",
+				le32_to_cpu(queue->recv_ddgst),
+				le32_to_cpu(queue->exp_ddgst));
+			return -EIO;
+		}
 	}
 
 	if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
Signature · 1 line
2.24.1
maintainer
#26 lore raw

Re: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule

Quoted 3 lines

crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.

Can you clarify in the patch description that this is only for pdu data digest and not header digest?

Quoted 5 lines

This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.

??

From my scan it doeesn't look like you do that.. Am I missing something? Can you explain?

Quoted 22 lines
---
  drivers/nvme/host/tcp.c | 66 ++++++++++++++++++++++++++++++++++++-----
  1 file changed, 58 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +1 -0
index 7bd97f856677..9a620d1dacb4 100644
@@ -94,6 +94,7 @@ struct nvme_tcp_queue {
  	size_t			data_remaining;
  	size_t			ddgst_remaining;
  	unsigned int		nr_cqe;
+	bool			crc_valid;
 
  	/* send state */
  	struct nvme_tcp_request *request;

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 237

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
@@ -233,6 +234,41 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
  	return nvme_tcp_pdu_data_left(req) <= len;
  }
 
+static inline bool nvme_tcp_device_ddgst_ok(struct nvme_tcp_queue *queue)

Maybe call it nvme_tcp_ddp_ddgst_ok?

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 243

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+{
+	return queue->crc_valid;
+}
+
+static inline void nvme_tcp_device_ddgst_update(struct nvme_tcp_queue *queue,
+						struct sk_buff *skb)

Maybe call it nvme_tcp_ddp_ddgst_update?

drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 254

Quoted 11 lines
drivers/nvme/host/tcp.c +11 -0
+{
+	if (queue->crc_valid)
+#ifdef CONFIG_TCP_DDP_CRC
+		queue->crc_valid = skb->ddp_crc;
+#else
+		queue->crc_valid = false;
+#endif
+}
+
+static void nvme_tcp_crc_recalculate(struct nvme_tcp_queue *queue,
+				     struct nvme_tcp_data_pdu *pdu)

Maybe call it nvme_tcp_ddp_ddgst_recalc?

Quoted 29 lines
drivers/nvme/host/tcp.c +18 -0
+{
+	struct nvme_tcp_request *req;
+	struct request *rq;
+
+	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+	if (!rq)
+		return;
+	req = blk_mq_rq_to_pdu(rq);
+	crypto_ahash_init(queue->rcv_hash);
+	req->ddp.sg_table.sgl = req->ddp.first_sgl;
+	/* req->ddp.sg_table is allocated and filled in nvme_tcp_setup_ddp */
+	ahash_request_set_crypt(queue->rcv_hash, req->ddp.sg_table.sgl, NULL,
+				le32_to_cpu(pdu->data_length));
+	crypto_ahash_update(queue->rcv_hash);
+}
+
+
  #ifdef CONFIG_TCP_DDP
 
  bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
@@ -706,6 +742,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
  	queue->pdu_offset = 0;
  	queue->data_remaining = -1;
  	queue->ddgst_remaining = 0;
+	queue->crc_valid = true;
  }
 
  static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)

drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 996

Quoted 5 lines
drivers/nvme/host/tcp.c +2 -0
@@ -955,6 +992,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  	struct nvme_tcp_request *req;
  	struct request *rq;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_device_ddgst_update(queue, skb);

Is the queue->data_digest condition missing here?

drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1036

Quoted 11 lines
drivers/nvme/host/tcp.c +1 -1
  	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
  	if (!rq) {
  		dev_err(queue->ctrl->ctrl.device,
@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  		recv_len = min_t(size_t, recv_len,
  				iov_iter_count(&req->iter));
 
-		if (queue->data_digest)
+		if (queue->data_digest && !test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
  			ret = skb_copy_and_hash_datagram_iter(skb, *offset,
  				&req->iter, recv_len, queue->rcv_hash);

This is the skb copy and hash, not clear why you say that you move this to the end...

drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1015

Quoted 6 lines
drivers/nvme/host/tcp.c +0 -1
  		else
@@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
  	if (!queue->data_remaining) {
  		if (queue->data_digest) {
-			nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);

If I instead do:

			if (!test_bit(NVME_TCP_Q_OFFLOADS,
                                      &queue->flags))
				nvme_tcp_ddgst_final(queue->rcv_hash,
						     &queue->exp_ddgst);

Does that help the mess in nvme_tcp_recv_ddgst?

Quoted 29 lines
drivers/nvme/host/tcp.c +7 -6
  			queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
  		} else {
  			if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
  	char *ddgst = (char *)&queue->recv_ddgst;
  	size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining);
  	off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
+	bool ddgst_offload_fail;
  	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_device_ddgst_update(queue, skb);
  	ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len);
  	if (unlikely(ret))
  		return ret;
@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
  	if (queue->ddgst_remaining)
  		return 0;
 
-	if (queue->recv_ddgst != queue->exp_ddgst) {
-		dev_err(queue->ctrl->ctrl.device,
-			"data digest error: recv %#x expected %#x\n",
-			le32_to_cpu(queue->recv_ddgst),
-			le32_to_cpu(queue->exp_ddgst));
-		return -EIO;
+	ddgst_offload_fail = !nvme_tcp_device_ddgst_ok(queue);
+	if (!test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) ||
+	    ddgst_offload_fail) {
+		if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&

drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1102

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+		    ddgst_offload_fail)
+			nvme_tcp_crc_recalculate(queue, pdu);
+
+		nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
+		if (queue->recv_ddgst != queue->exp_ddgst) {
+			dev_err(queue->ctrl->ctrl.device,
+				"data digest error: recv %#x expected %#x\n",
+				le32_to_cpu(queue->recv_ddgst),
+				le32_to_cpu(queue->exp_ddgst));
+			return -EIO;

This gets convoluted here...

Quoted 5 lines
drivers/nvme/host/tcp.c +1 -0
+		}
  	}
 
  	if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {

A message that is not in the archive (sent off-list or not yet received): PH0PR18MB3845764B48FD24C87FA34304CCED0@PH0PR18MB3845.namprd18.prod.outlook.com

A message that is not in the archive (sent off-list or not yet received): PH0PR18MB38458FD325BD77983D2623D4CCEB0@PH0PR18MB3845.namprd18.prod.outlook.com

#29 lore raw

RE: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule

Quoted 8 lines · Sagi Grimberg
Quoted 2 lines · Boris Pismenny

crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.

Can you clarify in the patch description that this is only for pdu data digest and not header digest?

Not a security expert, but according to my understanding, the NVMeTCP data digest is a layer 5 CRC, and as such it is expected to be end-to-end, meaning it is computed by layer 5 on the transmitter and verified on layer 5 on the receiver.
Any data corruption which happens in any of the lower layers, including their software processing, should be protected by this CRC. For example, if the IP or TCP stack has a bug that corrupts the NVMeTCP payload data, the CRC should protect against it. It seems that may not be the case with this offload.

Quoted 21 lines
Quoted 5 lines

This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.

??

From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?

Quoted 7 lines
---
  drivers/nvme/host/tcp.c | 66

++++++++++++++++++++++++++++++++++++-----

Quoted 2 lines
  1 file changed, 58 insertions(+), 8 deletions(-)
Quoted 9 lines

drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 97

Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0 -0

7bd97f856677..9a620d1dacb4 100644

drivers/nvme/host/tcp.c +1 -0
@@ -94,6 +94,7 @@ struct nvme_tcp_queue {
  	size_t			data_remaining;
  	size_t			ddgst_remaining;
  	unsigned int		nr_cqe;
+	bool			crc_valid;

I suggest to rename it to ddgst_valid.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#30 lore raw

Re: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule

Quoted 10 lines · Shai Malin
Quoted 7 lines · Sagi Grimberg
Quoted 2 lines · Boris Pismenny

crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.

Can you clarify in the patch description that this is only for pdu data digest and not header digest?

Not a security expert, but according to my understanding, the NVMeTCP data digest is a layer 5 CRC, and as such it is expected to be end-to-end, meaning it is computed by layer 5 on the transmitter and verified on layer 5 on the receiver.
Any data corruption which happens in any of the lower layers, including their software processing, should be protected by this CRC. For example, if the IP or TCP stack has a bug that corrupts the NVMeTCP payload data, the CRC should protect against it. It seems that may not be the case with this offload.

If the TCP/IP stack corrupts packet data, then likely many TCP/IP consumers will be effected, and it will be fixed promptly. Unlike with TOE, these bugs can be easily fixed/hotpatched by the community.

Quoted 19 lines
Quoted 19 lines
Quoted 5 lines

This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.

??

From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?

Quoted 6 lines
---
  drivers/nvme/host/tcp.c | 66

++++++++++++++++++++++++++++++++++++-----

Quoted 2 lines
  1 file changed, 58 insertions(+), 8 deletions(-)
Quoted 11 lines
Quoted 9 lines

drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 97

Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0 -0

7bd97f856677..9a620d1dacb4 100644

drivers/nvme/host/tcp.c +1 -0
@@ -94,6 +94,7 @@ struct nvme_tcp_queue {
  	size_t			data_remaining;
  	size_t			ddgst_remaining;
  	unsigned int		nr_cqe;
+	bool			crc_valid;

I suggest to rename it to ddgst_valid.

Sure

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#31 lore raw

Re: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule

In reply to Sagi Grimberg
Quoted 4 lines · Sagi Grimberg
Quoted 2 lines

crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.

Can you clarify in the patch description that this is only for pdu data digest and not header digest?

Will do

Quoted 9 lines
Quoted 5 lines

This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.

??

From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?

The performance of CRC data digest in the offload's fallback path may be less good compared to CRC calculation with skb_copy_and_hash.
To be clear, the fallback path is occurs when queue->data_digest && test_bit(NVME_TCP_Q_OFF_CRC_RX, &queue->flags), while we receive SKBs where skb->ddp_crc = 0

Quoted 8 lines
Quoted 8 lines
  	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
  	if (!rq) {
  		dev_err(queue->ctrl->ctrl.device,
drivers/nvme/host/tcp.c +0 -0
@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
  		recv_len = min_t(size_t, recv_len,
  				iov_iter_count(&req->iter));
 
Quoted 6 lines

drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1036

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -1
-		if (queue->data_digest)
+		if (queue->data_digest && !test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
  			ret = skb_copy_and_hash_datagram_iter(skb, *offset,
  				&req->iter, recv_len, queue->rcv_hash);

This is the skb copy and hash, not clear why you say that you move this to the end...

See the offload fallback path below

Quoted 8 lines

drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1015

Quoted 7 lines
drivers/nvme/host/tcp.c +0 -1
  		else
@@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
  	if (!queue->data_remaining) {
  		if (queue->data_digest) {
-			nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);

If I instead do:

Quoted 6 lines
			if (!test_bit(NVME_TCP_Q_OFFLOADS,
                                       &queue->flags))
				nvme_tcp_ddgst_final(queue->rcv_hash,
						     &queue->exp_ddgst);

Does that help the mess in nvme_tcp_recv_ddgst?

Not really, as the code path there takes care of the fallback path, i.e. offloaded requested, but didn't succeed.

Quoted 33 lines
Quoted 33 lines
drivers/nvme/host/tcp.c +10 -6
  			queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
  		} else {
  			if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
  	char *ddgst = (char *)&queue->recv_ddgst;
  	size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining);
  	off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
+	bool ddgst_offload_fail;
  	int ret;
 
+	if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags))
+		nvme_tcp_device_ddgst_update(queue, skb);
  	ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len);
  	if (unlikely(ret))
  		return ret;
@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
  	if (queue->ddgst_remaining)
  		return 0;
 
-	if (queue->recv_ddgst != queue->exp_ddgst) {
-		dev_err(queue->ctrl->ctrl.device,
-			"data digest error: recv %#x expected %#x\n",
-			le32_to_cpu(queue->recv_ddgst),
-			le32_to_cpu(queue->exp_ddgst));
-		return -EIO;
+	ddgst_offload_fail = !nvme_tcp_device_ddgst_ok(queue);
+	if (!test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) ||
+	    ddgst_offload_fail) {
+		if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
+		    ddgst_offload_fail)
+			nvme_tcp_crc_recalculate(queue, pdu);
+
Quoted 8 lines

drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1102

Quoted 7 lines
drivers/nvme/host/tcp.c +7 -0
+		nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
+		if (queue->recv_ddgst != queue->exp_ddgst) {
+			dev_err(queue->ctrl->ctrl.device,
+				"data digest error: recv %#x expected %#x\n",
+				le32_to_cpu(queue->recv_ddgst),
+				le32_to_cpu(queue->exp_ddgst));
+			return -EIO;

This gets convoluted here...

Will try to simplify, the general idea is that there are 3 paths with common code:
1. non-offload
2. offload failed
3. offload success
(1) and (2) share the code for finalizing checking the data digest, while (3) skips this entirely.

In other words, how about this:

```
          offload_fail = !nvme_tcp_ddp_ddgst_ok(queue);                                               
          offload = test_bit(NVME_TCP_Q_OFF_CRC_RX, &queue->flags);                                   
          if (!offload || offload_fail) {                                                             
                  if (offload && offload_fail) // software-fallback                                   
                          nvme_tcp_ddp_ddgst_recalc(queue, pdu);                                      
                                                                                                      
                  nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);                           
                  if (queue->recv_ddgst != queue->exp_ddgst) {                                        
                          dev_err(queue->ctrl->ctrl.device,                                           
                                  "data digest error: recv %#x expected %#x\n",                       
                                  le32_to_cpu(queue->recv_ddgst),                                     
                                  le32_to_cpu(queue->exp_ddgst));                                     
                          return -EIO;                                                                
                  }                                                                                   
          } 
```
Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/tcp.c +1 -0
+		}
  	}
 
  	if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
#32 lore raw

RFC PATCH 8/10 nvme-tcp: Deal with netdevice DOWN events

In reply to Boris Pismenny

From: Or Gerlitz <[email protected]>

For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.

These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroyes them.

Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.

We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.

Since the recovery flow runs in a separate/dedicated WQ we need to wait in the notifier code for an ACK that all offloaded queues were stopped which means that the teardown queue offload ndo was called and the NIC doesn't have any resources related to that connection any more.

This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.

---
 drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +37 -2
index 9a620d1dacb4..7569b47f0414 100644
@@ -144,6 +144,7 @@ struct nvme_tcp_ctrl {
 
 static LIST_HEAD(nvme_tcp_ctrl_list);
 static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
+static struct notifier_block nvme_tcp_netdevice_nb;
 static struct workqueue_struct *nvme_tcp_wq;
 static const struct blk_mq_ops nvme_tcp_mq_ops;
 static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
 		queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
 		queue->ctrl->ctrl.max_hw_sectors =
 			limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
-	} else {
-		queue->ctrl->offloading_netdev = NULL;
 	}
 
 	dev_put(netdev);
@@ -1992,6 +1991,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
 {
 	int ret;
 
+	to_tcp_ctrl(ctrl)->offloading_netdev = NULL;
+
 	ret = nvme_tcp_alloc_queue(ctrl, 0, NVME_AQ_DEPTH);
 	if (ret)
 		return ret;
@@ -2885,6 +2886,26 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
 	return ERR_PTR(ret);
 }
 
+static int nvme_tcp_netdev_event(struct notifier_block *this,
+				 unsigned long event, void *ptr)
+{
+	struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
+	struct nvme_tcp_ctrl *ctrl;
+
+	switch (event) {
+	case NETDEV_GOING_DOWN:
+		mutex_lock(&nvme_tcp_ctrl_mutex);
+		list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) {
+			if (ndev != ctrl->offloading_netdev)
+				continue;
+			nvme_tcp_error_recovery(&ctrl->ctrl);
+		}
+		mutex_unlock(&nvme_tcp_ctrl_mutex);
+		flush_workqueue(nvme_reset_wq);
+	}
+	return NOTIFY_DONE;
+}
+
 static struct nvmf_transport_ops nvme_tcp_transport = {
 	.name		= "tcp",
 	.module		= THIS_MODULE,
@@ -2899,13 +2920,26 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
 
 static int __init nvme_tcp_init_module(void)
 {
+	int ret;
+
 	nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq",
 			WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
 	if (!nvme_tcp_wq)
 		return -ENOMEM;
 
+	nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event;
+	ret = register_netdevice_notifier(&nvme_tcp_netdevice_nb);
+	if (ret) {
+		pr_err("failed to register netdev notifier\n");
+		goto out_err_reg_notifier;
+	}
+
 	nvmf_register_transport(&nvme_tcp_transport);
 	return 0;
+
+out_err_reg_notifier:
+	destroy_workqueue(nvme_tcp_wq);
+	return ret;
 }
 
 static void __exit nvme_tcp_cleanup_module(void)
@@ -2913,6 +2947,7 @@ static void __exit nvme_tcp_cleanup_module(void)
 	struct nvme_tcp_ctrl *ctrl;
 
 	nvmf_unregister_transport(&nvme_tcp_transport);
+	unregister_netdevice_notifier(&nvme_tcp_netdevice_nb);
 
 	mutex_lock(&nvme_tcp_ctrl_mutex);
 	list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list)
Signature · 1 line
2.24.1
maintainer
#33 lore raw

Re: [PATCH net-next RFC v1 08/10] nvme-tcp: Deal with netdevice DOWN events

Quoted 47 lines · Boris Pismenny

From: Or Gerlitz <[email protected]>

For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.

These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroyes them.

Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.

We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.

Since the recovery flow runs in a separate/dedicated WQ we need to wait in the notifier code for an ACK that all offloaded queues were stopped which means that the teardown queue offload ndo was called and the NIC doesn't have any resources related to that connection any more.

This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.

---
  drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++--
  1 file changed, 37 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +1 -0
index 9a620d1dacb4..7569b47f0414 100644
@@ -144,6 +144,7 @@ struct nvme_tcp_ctrl {
 
  static LIST_HEAD(nvme_tcp_ctrl_list);
  static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
+static struct notifier_block nvme_tcp_netdevice_nb;
  static struct workqueue_struct *nvme_tcp_wq;
  static const struct blk_mq_ops nvme_tcp_mq_ops;
  static const struct blk_mq_ops nvme_tcp_admin_mq_ops;

drivers/nvme/host/tcp.c int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, · line 416

Quoted 5 lines
drivers/nvme/host/tcp.c +0 -2
@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
  		queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
  		queue->ctrl->ctrl.max_hw_sectors =
  			limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
-	} else {
-		queue->ctrl->offloading_netdev = NULL;

Squash this change to the patch that introduced it.

Quoted 22 lines
drivers/nvme/host/tcp.c +8 -0
  	}
 
  	dev_put(netdev);
@@ -1992,6 +1991,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
  {
  	int ret;
 
+	to_tcp_ctrl(ctrl)->offloading_netdev = NULL;
+
  	ret = nvme_tcp_alloc_queue(ctrl, 0, NVME_AQ_DEPTH);
  	if (ret)
  		return ret;
@@ -2885,6 +2886,26 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
  	return ERR_PTR(ret);
  }
 
+static int nvme_tcp_netdev_event(struct notifier_block *this,
+				 unsigned long event, void *ptr)
+{
+	struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
+	struct nvme_tcp_ctrl *ctrl;
+

drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 2904

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+	switch (event) {
+	case NETDEV_GOING_DOWN:
+		mutex_lock(&nvme_tcp_ctrl_mutex);
+		list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) {
+			if (ndev != ctrl->offloading_netdev)
+				continue;
+			nvme_tcp_error_recovery(&ctrl->ctrl);
+		}
+		mutex_unlock(&nvme_tcp_ctrl_mutex);
+		flush_workqueue(nvme_reset_wq);

Worth a small comment that this we want the err_work to complete here. So if someone changes workqueues he may see this.

Quoted 7 lines
drivers/nvme/host/tcp.c +4 -0
+	}
+	return NOTIFY_DONE;
+}
+
  static struct nvmf_transport_ops nvme_tcp_transport = {
  	.name		= "tcp",
  	.module		= THIS_MODULE,
#34 lore raw

Re: [PATCH net-next RFC v1 08/10] nvme-tcp: Deal with netdevice DOWN events

Quoted 8 lines · Sagi Grimberg

drivers/nvme/host/tcp.c int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, · line 416

Quoted 6 lines
drivers/nvme/host/tcp.c +0 -2
@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue,
              queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len;
              queue->ctrl->ctrl.max_hw_sectors =
                      limits->max_ddp_sgl_len << (ilog2(SZ_4K) - 9);
-     } else {
-             queue->ctrl->offloading_netdev = NULL;

Squash this change to the patch that introduced it.

OK, will look on that and I guess it should be fine to make this as you suggested

Quoted 10 lines

drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 2904

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+     case NETDEV_GOING_DOWN:
+             mutex_lock(&nvme_tcp_ctrl_mutex);
+             list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) {
+                     if (ndev != ctrl->offloading_netdev)
+                             continue;
+                     nvme_tcp_error_recovery(&ctrl->ctrl);
+             }
+             mutex_unlock(&nvme_tcp_ctrl_mutex);
+             flush_workqueue(nvme_reset_wq);
Quoted 2 lines

Worth a small comment that this we want the err_work to complete here. So if someone changes workqueues he may see this.

ack

#35 lore raw

RFC PATCH 9/10 net/mlx5e: Add NVMEoTCP offload

In reply to Boris Pismenny

Add NVMEoTCP offload to mlx5.

This patch implements the NVME-TCP offload added by previous commits. Similarly to other layer-5 offload, the offload is mapped to a TIR, and updated using WQEs.

- Use 128B CQEs when NVME-TCP offload is enabled
- Implement asynchronous ddp invalidaation by completing the nvme-tcp request only when the invalidate UMR is done
- Use KLM UMRs to implement ddp
- Use a dedicated icosq for all NVME-TCP work. This SQ is unique in the sense that it is driven directly by the NVME-TCP layer to submit and invalidate ddp requests.
- Add statistics for offload packets/bytes, ddp setup/teardown, and queue init/teardown

---
 .../net/ethernet/mellanox/mlx5/core/Kconfig   |  11 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  11 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |   4 +
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |  13 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |   9 +-
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      |  10 +
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 894 ++++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    | 116 +++
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |  79 ++
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  73 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  25 +-
 .../ethernet/mellanox/mlx5/core/en_stats.c    |  26 +
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  16 +
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |   6 +
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   6 +
 include/linux/mlx5/device.h                   |   8 +
 include/linux/mlx5/mlx5_ifc.h                 | 121 ++-
 include/linux/mlx5/qp.h                       |   1 +
 19 files changed, 1419 insertions(+), 12 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +11 -0
index 99f1ec3b2575..20fc7795f7c6 100644
@@ -201,3 +201,14 @@ config MLX5_SW_STEERING
 	default y
 	help
 	Build support for software-managed steering in the NIC.
+
+config MLX5_EN_NVMEOTCP
+	bool "NVMEoTCP accelaration"
+	depends on MLX5_CORE_EN
+	depends on TCP_DDP
+	depends on TCP_DDP_CRC
+	default y
+	help
+	Build support for NVMEoTCP accelaration in the NIC.
+	Note: Support for hardware with this capability needs to be selected
+	for this option to become available.
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -0
index 9826a041e407..9dd6b41c2486 100644
@@ -84,3 +84,5 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o
 					steering/dr_ste.o steering/dr_send.o \
 					steering/dr_cmd.o steering/dr_fw.o \
 					steering/dr_action.o steering/fs_dr.o
+
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +10 -1
index 5368e06cd71c..a8c0fc98b394 100644
@@ -209,7 +209,10 @@ struct mlx5e_umr_wqe {
 	struct mlx5_wqe_ctrl_seg       ctrl;
 	struct mlx5_wqe_umr_ctrl_seg   uctrl;
 	struct mlx5_mkey_seg           mkc;
-	struct mlx5_mtt                inline_mtts[0];
+	union {
+		struct mlx5_mtt        inline_mtts[0];
+		struct mlx5_klm        inline_klms[0];
+	};
 };
 
 extern const char mlx5e_self_tests[][ETH_GSTRING_LEN];
@@ -642,6 +645,9 @@ struct mlx5e_channel {
 	struct mlx5e_xdpsq         rq_xdpsq;
 	struct mlx5e_txqsq         sq[MLX5E_MAX_NUM_TC];
 	struct mlx5e_icosq         icosq;   /* internal control operations */
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct mlx5e_icosq         nvmeotcpsq;   /* nvmeotcp umrs  */
+#endif
 	bool                       xdp;
 	struct napi_struct         napi;
 	struct device             *pdev;
@@ -815,6 +821,9 @@ struct mlx5e_priv {
 #endif
 #ifdef CONFIG_MLX5_EN_TLS
 	struct mlx5e_tls          *tls;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct mlx5e_nvmeotcp      *nvmeotcp;
 #endif
 	struct devlink_health_reporter *tx_reporter;
 	struct devlink_health_reporter *rx_reporter;
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +4 -0
index a87273e801b2..7d280c35f538 100644
@@ -16,6 +16,7 @@ struct mlx5e_cq_param {
 	struct mlx5_wq_param       wq;
 	u16                        eq_ix;
 	u8                         cq_period_mode;
+	bool                       force_cqe128;
 };
 
 struct mlx5e_rq_param {
@@ -38,6 +39,9 @@ struct mlx5e_channel_param {
 	struct mlx5e_sq_param      xdp_sq;
 	struct mlx5e_sq_param      icosq;
 	struct mlx5e_sq_param      async_icosq;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	struct mlx5e_sq_param      nvmeotcpsq;
+#endif
 };
 
 static inline bool mlx5e_qid_get_ch_if_in_group(struct mlx5e_params *params,
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +13 -0
index 07ee1d236ab3..1aec1900bee9 100644
@@ -32,6 +32,11 @@ enum mlx5e_icosq_wqe_type {
 	MLX5E_ICOSQ_WQE_SET_PSV_TLS,
 	MLX5E_ICOSQ_WQE_GET_PSV_TLS,
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	MLX5E_ICOSQ_WQE_UMR_NVME_TCP,
+	MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE,
+	MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP,
+#endif
 };
 
 /* General */
@@ -173,6 +178,14 @@ struct mlx5e_icosq_wqe_info {
 		struct {
 			struct mlx5e_ktls_rx_resync_buf *buf;
 		} tls_get_params;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+		struct {
+			struct mlx5e_nvmeotcp_queue *queue;
+		} nvmeotcp_q;
+		struct {
+			struct nvmeotcp_queue_entry *entry;
+		} nvmeotcp_qe;
 #endif
 	};
 };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +8 -1
index 2ea1cdc1ca54..2fa6f9286ed9 100644
@@ -39,6 +39,7 @@
 #include "en_accel/ipsec_rxtx.h"
 #include "en_accel/tls.h"
 #include "en_accel/tls_rxtx.h"
+#include "en_accel/nvmeotcp.h"
 #include "en.h"
 #include "en/txrx.h"
 
@@ -162,11 +163,17 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,
 
 static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv)
 {
-	return mlx5e_ktls_init_rx(priv);
+	int tls, nvmeotcp;
+
+	tls = mlx5e_ktls_init_rx(priv);
+	nvmeotcp = mlx5e_nvmeotcp_init_rx(priv);
+
+	return tls && nvmeotcp;
 }
 
 static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv)
 {
+	mlx5e_nvmeotcp_cleanup_rx(priv);
 	mlx5e_ktls_cleanup_rx(priv);
 }
 #endif /* __MLX5E_EN_ACCEL_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +10 -0
index 97f1594cee11..feded6c8cca1 100644
@@ -14,6 +14,7 @@ enum accel_fs_tcp_type {
 struct mlx5e_accel_fs_tcp {
 	struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES];
 	struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES];
+	refcount_t		ref_count;
 };
 
 static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
@@ -335,6 +336,7 @@ static int accel_fs_tcp_enable(struct mlx5e_priv *priv)
 			return err;
 		}
 	}
+	refcount_set(&priv->fs.accel_tcp->ref_count, 1);
 	return 0;
 }
 
@@ -358,6 +360,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)
 	if (!priv->fs.accel_tcp)
 		return;
 
+	if (!refcount_dec_and_test(&priv->fs.accel_tcp->ref_count))
+		return;
+
 	accel_fs_tcp_disable(priv);
 
 	for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++)
@@ -374,6 +379,11 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)
 	if (!MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, ft_field_support.outer_ip_version))
 		return -EOPNOTSUPP;
 
+	if (priv->fs.accel_tcp) {
+		refcount_inc(&priv->fs.accel_tcp->ref_count);
+		return 0;
+	}
+
 	priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
 	if (!priv->fs.accel_tcp)
 		return -ENOMEM;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c new file +894 -0

894 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h new file +116 -0

116 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h new file +79 -0

79 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_main.c +69 -4

185 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +23 -2
index 599f5b5ebc97..ac99dbb3573a 100644
@@ -47,6 +47,7 @@
 #include "fpga/ipsec.h"
 #include "en_accel/ipsec_rxtx.h"
 #include "en_accel/tls_rxtx.h"
+#include "en_accel/nvmeotcp.h"
 #include "lib/clock.h"
 #include "en/xdp.h"
 #include "en/xsk/rx.h"
@@ -617,16 +618,26 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
 		ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc);
 		wi = &sq->db.wqe_info[ci];
 		sqcc += wi->num_wqebbs;
-#ifdef CONFIG_MLX5_EN_TLS
 		switch (wi->wqe_type) {
+#ifdef CONFIG_MLX5_EN_TLS
 		case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
 			mlx5e_ktls_handle_ctx_completion(wi);
 			break;
 		case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
 			mlx5e_ktls_handle_get_psv_completion(wi, sq);
 			break;
-		}
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
+			break;
+		case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE:
+			mlx5e_nvmeotcp_ddp_inv_done(wi);
+			break;
+		case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
+			mlx5e_nvmeotcp_ctx_comp(wi);
+			break;
+#endif
+		}
 	}
 	sq->cc = sqcc;
 }
@@ -695,6 +706,16 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 			case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
 				mlx5e_ktls_handle_get_psv_completion(wi, sq);
 				break;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP:
+				break;
+			case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE:
+				mlx5e_nvmeotcp_ddp_inv_done(wi);
+				break;
+			case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP:
+				mlx5e_nvmeotcp_ctx_comp(wi);
+				break;
 #endif
 			default:
 				netdev_WARN_ONCE(cq->channel->netdev,
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +26 -0
index 78f6a6f0a7e0..25d203d64bb2 100644
@@ -34,6 +34,7 @@
 #include "en.h"
 #include "en_accel/tls.h"
 #include "en_accel/en_accel.h"
+#include "en_accel/nvmeotcp.h"
 
 static unsigned int stats_grps_num(struct mlx5e_priv *priv)
 {
@@ -189,6 +190,14 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_ok) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_skip) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) },
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init_fail) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_teardown) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) },
 #endif
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
@@ -314,6 +323,14 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
 		s->rx_tls_resync_res_ok     += rq_stats->tls_resync_res_ok;
 		s->rx_tls_resync_res_skip   += rq_stats->tls_resync_res_skip;
 		s->rx_tls_err               += rq_stats->tls_err;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+		s->rx_nvmeotcp_queue_init      += rq_stats->nvmeotcp_queue_init;
+		s->rx_nvmeotcp_queue_init_fail += rq_stats->nvmeotcp_queue_init_fail;
+		s->rx_nvmeotcp_queue_teardown  += rq_stats->nvmeotcp_queue_teardown;
+		s->rx_nvmeotcp_ddp_setup       += rq_stats->nvmeotcp_ddp_setup;
+		s->rx_nvmeotcp_ddp_setup_fail  += rq_stats->nvmeotcp_ddp_setup_fail;
+		s->rx_nvmeotcp_ddp_teardown    += rq_stats->nvmeotcp_ddp_teardown;
 #endif
 		s->ch_events      += ch_stats->events;
 		s->ch_poll        += ch_stats->poll;
@@ -390,6 +407,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
 			s->tx_tls_drop_no_sync_data += sq_stats->tls_drop_no_sync_data;
 			s->tx_tls_drop_bypass_req   += sq_stats->tls_drop_bypass_req;
 #endif
+
 			s->tx_cqes		+= sq_stats->cqes;
 
 			/* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92657 */
@@ -1559,6 +1577,14 @@ static const struct counter_desc rq_stats_desc[] = {
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_resync_res_skip) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) },
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init_fail) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_teardown) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) },
+#endif
 };
 
 static const struct counter_desc sq_stats_desc[] = {
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +16 -0
index 162daaadb0d8..5c1c0ad88ff4 100644
@@ -175,6 +175,14 @@ struct mlx5e_sw_stats {
 	u64 rx_congst_umr;
 	u64 rx_arfs_err;
 	u64 rx_recover;
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	u64 rx_nvmeotcp_queue_init;
+	u64 rx_nvmeotcp_queue_init_fail;
+	u64 rx_nvmeotcp_queue_teardown;
+	u64 rx_nvmeotcp_ddp_setup;
+	u64 rx_nvmeotcp_ddp_setup_fail;
+	u64 rx_nvmeotcp_ddp_teardown;
+#endif
 	u64 ch_events;
 	u64 ch_poll;
 	u64 ch_arm;
@@ -338,6 +346,14 @@ struct mlx5e_rq_stats {
 	u64 tls_resync_res_skip;
 	u64 tls_err;
 #endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	u64 nvmeotcp_queue_init;
+	u64 nvmeotcp_queue_init_fail;
+	u64 nvmeotcp_queue_teardown;
+	u64 nvmeotcp_ddp_setup;
+	u64 nvmeotcp_ddp_setup_fail;
+	u64 nvmeotcp_ddp_teardown;
+#endif
 };
 
 struct mlx5e_sq_stats {
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +6 -0
index d5868670f8a5..c2bd2e8d5508 100644
@@ -158,6 +158,9 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 		 * queueing more WQEs and overflowing the async ICOSQ.
 		 */
 		clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state);
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	mlx5e_poll_ico_cq(&c->nvmeotcpsq.cq);
+#endif
 
 	busy |= INDIRECT_CALL_2(rq->post_wqes,
 				mlx5e_post_rx_mpwqes,
@@ -196,6 +199,9 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	mlx5e_cq_arm(&rq->cq);
 	mlx5e_cq_arm(&c->icosq.cq);
 	mlx5e_cq_arm(&c->async_icosq.cq);
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+	mlx5e_cq_arm(&c->nvmeotcpsq.cq);
+#endif
 	mlx5e_cq_arm(&c->xdpsq.cq);
 
 	if (xsk_open) {
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
index 02558ac2ace6..5e7544ccae91 100644
@@ -256,6 +256,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 			return err;
 	}
 
+	if (MLX5_CAP_GEN(dev, nvmeotcp)) {
+		err = mlx5_core_get_caps(dev, MLX5_CAP_DEV_NVMEOTCP);
+		if (err)
+			return err;
+	}
+
 	return 0;
 }
 
include/linux/mlx5/device.h +8 -0
index 81ca5989009b..afadf4cf6d7a 100644
@@ -263,6 +263,7 @@ enum {
 enum {
 	MLX5_MKEY_MASK_LEN		= 1ull << 0,
 	MLX5_MKEY_MASK_PAGE_SIZE	= 1ull << 1,
+	MLX5_MKEY_MASK_XLT_OCT_SIZE	= 1ull << 2,
 	MLX5_MKEY_MASK_START_ADDR	= 1ull << 6,
 	MLX5_MKEY_MASK_PD		= 1ull << 7,
 	MLX5_MKEY_MASK_EN_RINVAL	= 1ull << 8,
@@ -1153,6 +1154,7 @@ enum mlx5_cap_type {
 	MLX5_CAP_VDPA_EMULATION = 0x13,
 	MLX5_CAP_DEV_EVENT = 0x14,
 	MLX5_CAP_IPSEC,
+	MLX5_CAP_DEV_NVMEOTCP = 0x19,
 	/* NUM OF CAP Types */
 	MLX5_CAP_NUM
 };
@@ -1373,6 +1375,12 @@ enum mlx5_qcam_feature_groups {
 #define MLX5_CAP_IPSEC(mdev, cap)\
 	MLX5_GET(ipsec_cap, (mdev)->caps.hca_cur[MLX5_CAP_IPSEC], cap)
 
+#define MLX5_CAP_DEV_NVMEOTCP(mdev, cap)\
+	MLX5_GET(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap)
+
+#define MLX5_CAP64_NVMEOTCP(mdev, cap)\
+	MLX5_GET64(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap)
+
 enum {
 	MLX5_CMD_STAT_OK			= 0x0,
 	MLX5_CMD_STAT_INT_ERR			= 0x1,
include/linux/mlx5/mlx5_ifc.h +117 -4

184 lines not on this page · raw message

include/linux/mlx5/qp.h +1 -0
index 36492a1342cf..8b62d3f4a868 100644
@@ -220,6 +220,7 @@ struct mlx5_wqe_ctrl_seg {
 #define MLX5_WQE_CTRL_OPCODE_MASK 0xff
 #define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00
 #define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8
+#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8
 
 enum {
 	MLX5_ETH_WQE_L3_INNER_CSUM      = 1 << 4,
Signature · 1 line
2.24.1
#36 lore raw

Re: [PATCH net-next RFC v1 09/10] net/mlx5e: Add NVMEoTCP offload

reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…

Hi Boris,

[FYI, it's a private test report for your RFC patch.] [auto build test ERROR on net-next/master]

url:    https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1
config: xtensa-allyesconfig (attached as .config)
compiler: xtensa-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/990254a46f72010830cdf80ed80ed506f268e759
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 git checkout 990254a46f72010830cdf80ed80ed506f268e759
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=xtensa

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

In file included from include/linux/bits.h:6,
from include/linux/bitops.h:5,
from include/linux/kernel.h:12,
from include/linux/list.h:9,
from include/linux/timer.h:5,
from include/linux/netdevice.h:24,
from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:4:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]

       7 | #define BIT(nr)   (UL(1) << (nr))
         |                          ^~
   include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
   10680 |  MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
         |                                                                ^~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:68:6: error: redefinition of 'mlx5e_accel_fs_del_sk'
      68 | void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
         |      ^~~~~~~~~~~~~~~~~~~~~
   In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:23:20: note: previous definition of 'mlx5e_accel_fs_del_sk' was here
      23 | static inline void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule) {}
         |                    ^~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:73:26: error: redefinition of 'mlx5e_accel_fs_add_sk'
      73 | struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
         |                          ^~~~~~~~~~~~~~~~~~~~~
   In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:19:40: note: previous definition of 'mlx5e_accel_fs_add_sk' was here
      19 | static inline struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
         |                                        ^~~~~~~~~~~~~~~~~~~~~
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_add_sk':
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:88:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
      88 |  fs_tcp = priv->fs.accel_tcp;
         |                   ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_add_default_rule':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:160:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     160 |  fs_tcp = priv->fs.accel_tcp;
         |                   ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_create_table':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:269:41: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     269 |  struct mlx5e_flow_table *ft = &priv->fs.accel_tcp->tables[type];
         |                                         ^
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:276:18: error: 'MLX5E_ACCEL_FS_TCP_FT_LEVEL' undeclared (first use in this function); did you mean 'MLX5E_ACCEL_FS_ESP_FT_LEVEL'?
     276 |  ft_attr.level = MLX5E_ACCEL_FS_TCP_FT_LEVEL;
         |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
         |                  MLX5E_ACCEL_FS_ESP_FT_LEVEL
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:276:18: note: each undeclared identifier is reported only once for each function it appears in
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_enable':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:328:21: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     328 |   dest.ft = priv->fs.accel_tcp->tables[i].t;
         |                     ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:339:24: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     339 |  refcount_set(&priv->fs.accel_tcp->ref_count, 1);
         |                        ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_destroy_table':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:347:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     347 |  fs_tcp = priv->fs.accel_tcp;
         |                   ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: At top level:
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:356:6: error: redefinition of 'mlx5e_accel_fs_tcp_destroy'
     356 | void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)
         |      ^~~~~~~~~~~~~~~~~~~~~~~~~~
   In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:18:20: note: previous definition of 'mlx5e_accel_fs_tcp_destroy' was here
      18 | static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
         |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_tcp_destroy':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:360:15: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     360 |  if (!priv->fs.accel_tcp)
         |               ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:363:38: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     363 |  if (!refcount_dec_and_test(&priv->fs.accel_tcp->ref_count))
         |                                      ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:371:16: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     371 |  kfree(priv->fs.accel_tcp);
         |                ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:372:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     372 |  priv->fs.accel_tcp = NULL;
         |          ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: At top level:
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:375:5: error: redefinition of 'mlx5e_accel_fs_tcp_create'
     375 | int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)
         |     ^~~~~~~~~~~~~~~~~~~~~~~~~
   In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:17:19: note: previous definition of 'mlx5e_accel_fs_tcp_create' was here
      17 | static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
         |                   ^~~~~~~~~~~~~~~~~~~~~~~~~
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_tcp_create':
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:382:14: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     382 |  if (priv->fs.accel_tcp) {
         |              ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:383:25: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     383 |   refcount_inc(&priv->fs.accel_tcp->ref_count);
         |                         ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:387:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     387 |  priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
         |          ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:387:47: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     387 |  priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
         |                                               ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:388:15: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     388 |  if (!priv->fs.accel_tcp)
         |               ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:407:16: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     407 |  kfree(priv->fs.accel_tcp);
         |                ^
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:408:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
     408 |  priv->fs.accel_tcp = NULL;
         |          ^

--
In file included from include/linux/bits.h:6,
from include/linux/bitops.h:5,
from include/linux/kernel.h:12,
from include/linux/list.h:9,
from include/linux/timer.h:5,
from include/linux/netdevice.h:24,
from drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c:4:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]

       7 | #define BIT(nr)   (UL(1) << (nr))
         |                          ^~
   include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
   10680 |  MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
         |                                                                ^~~
   drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c: In function 'mlx5e_nvmeotcp_queue_init':
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c:613:7: warning: 'stats' is used uninitialized in this function [-Wuninitialized]
     613 |  stats->nvmeotcp_queue_init_fail++;
         |  ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/mlx5e_accel_fs_del_sk +68 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c

5229a96e59ec324 Tariq Toukan 2020-06-16 67

5229a96e59ec324 Tariq Toukan 2020-06-16  @68  void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
5229a96e59ec324 Tariq Toukan 2020-06-16   69  {
5229a96e59ec324 Tariq Toukan 2020-06-16   70  	mlx5_del_flow_rules(rule);
5229a96e59ec324 Tariq Toukan 2020-06-16   71  }
5229a96e59ec324 Tariq Toukan 2020-06-16   72  
5229a96e59ec324 Tariq Toukan 2020-06-16  @73  struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
5229a96e59ec324 Tariq Toukan 2020-06-16   74  					       struct sock *sk, u32 tirn,
5229a96e59ec324 Tariq Toukan 2020-06-16   75  					       uint32_t flow_tag)
5229a96e59ec324 Tariq Toukan 2020-06-16   76  {
5229a96e59ec324 Tariq Toukan 2020-06-16   77  	struct mlx5_flow_destination dest = {};
5229a96e59ec324 Tariq Toukan 2020-06-16   78  	struct mlx5e_flow_table *ft = NULL;
5229a96e59ec324 Tariq Toukan 2020-06-16   79  	struct mlx5e_accel_fs_tcp *fs_tcp;
5229a96e59ec324 Tariq Toukan 2020-06-16   80  	MLX5_DECLARE_FLOW_ACT(flow_act);
5229a96e59ec324 Tariq Toukan 2020-06-16   81  	struct mlx5_flow_handle *flow;
5229a96e59ec324 Tariq Toukan 2020-06-16   82  	struct mlx5_flow_spec *spec;
5229a96e59ec324 Tariq Toukan 2020-06-16   83  
5229a96e59ec324 Tariq Toukan 2020-06-16   84  	spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
5229a96e59ec324 Tariq Toukan 2020-06-16   85  	if (!spec)
5229a96e59ec324 Tariq Toukan 2020-06-16   86  		return ERR_PTR(-ENOMEM);
5229a96e59ec324 Tariq Toukan 2020-06-16   87  
5229a96e59ec324 Tariq Toukan 2020-06-16  @88  	fs_tcp = priv->fs.accel_tcp;
5229a96e59ec324 Tariq Toukan 2020-06-16   89  
5229a96e59ec324 Tariq Toukan 2020-06-16   90  	spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
5229a96e59ec324 Tariq Toukan 2020-06-16   91  
5229a96e59ec324 Tariq Toukan 2020-06-16   92  	switch (sk->sk_family) {
5229a96e59ec324 Tariq Toukan 2020-06-16   93  	case AF_INET:
5229a96e59ec324 Tariq Toukan 2020-06-16   94  		accel_fs_tcp_set_ipv4_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16   95  		ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16   96  		mlx5e_dbg(HW, priv, "%s flow is %pI4:%d -> %pI4:%d\n", __func__,
5229a96e59ec324 Tariq Toukan 2020-06-16   97  			  &inet_sk(sk)->inet_rcv_saddr,
5229a96e59ec324 Tariq Toukan 2020-06-16   98  			  inet_sk(sk)->inet_sport,
5229a96e59ec324 Tariq Toukan 2020-06-16   99  			  &inet_sk(sk)->inet_daddr,
5229a96e59ec324 Tariq Toukan 2020-06-16  100  			  inet_sk(sk)->inet_dport);
5229a96e59ec324 Tariq Toukan 2020-06-16  101  		break;
5229a96e59ec324 Tariq Toukan 2020-06-16  102  #if IS_ENABLED(CONFIG_IPV6)
5229a96e59ec324 Tariq Toukan 2020-06-16  103  	case AF_INET6:
5229a96e59ec324 Tariq Toukan 2020-06-16  104  		if (!sk->sk_ipv6only &&
5229a96e59ec324 Tariq Toukan 2020-06-16  105  		    ipv6_addr_type(&sk->sk_v6_daddr) == IPV6_ADDR_MAPPED) {
5229a96e59ec324 Tariq Toukan 2020-06-16  106  			accel_fs_tcp_set_ipv4_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16  107  			ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16  108  		} else {
5229a96e59ec324 Tariq Toukan 2020-06-16  109  			accel_fs_tcp_set_ipv6_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16  110  			ft = &fs_tcp->tables[ACCEL_FS_IPV6_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16  111  		}
5229a96e59ec324 Tariq Toukan 2020-06-16  112  		break;
5229a96e59ec324 Tariq Toukan 2020-06-16  113  #endif
5229a96e59ec324 Tariq Toukan 2020-06-16  114  	default:
5229a96e59ec324 Tariq Toukan 2020-06-16  115  		break;
5229a96e59ec324 Tariq Toukan 2020-06-16  116  	}
5229a96e59ec324 Tariq Toukan 2020-06-16  117  
5229a96e59ec324 Tariq Toukan 2020-06-16  118  	if (!ft) {
5229a96e59ec324 Tariq Toukan 2020-06-16  119  		flow = ERR_PTR(-EINVAL);
5229a96e59ec324 Tariq Toukan 2020-06-16  120  		goto out;
5229a96e59ec324 Tariq Toukan 2020-06-16  121  	}
5229a96e59ec324 Tariq Toukan 2020-06-16  122  
5229a96e59ec324 Tariq Toukan 2020-06-16  123  	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
5229a96e59ec324 Tariq Toukan 2020-06-16  124  			 outer_headers.tcp_dport);
5229a96e59ec324 Tariq Toukan 2020-06-16  125  	MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
5229a96e59ec324 Tariq Toukan 2020-06-16  126  			 outer_headers.tcp_sport);
5229a96e59ec324 Tariq Toukan 2020-06-16  127  	MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_dport,
5229a96e59ec324 Tariq Toukan 2020-06-16  128  		 ntohs(inet_sk(sk)->inet_sport));
5229a96e59ec324 Tariq Toukan 2020-06-16  129  	MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_sport,
5229a96e59ec324 Tariq Toukan 2020-06-16  130  		 ntohs(inet_sk(sk)->inet_dport));
5229a96e59ec324 Tariq Toukan 2020-06-16  131  
5229a96e59ec324 Tariq Toukan 2020-06-16  132  	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
5229a96e59ec324 Tariq Toukan 2020-06-16  133  	dest.tir_num = tirn;
5229a96e59ec324 Tariq Toukan 2020-06-16  134  	if (flow_tag != MLX5_FS_DEFAULT_FLOW_TAG) {
5229a96e59ec324 Tariq Toukan 2020-06-16  135  		spec->flow_context.flow_tag = flow_tag;
5229a96e59ec324 Tariq Toukan 2020-06-16  136  		spec->flow_context.flags = FLOW_CONTEXT_HAS_TAG;
5229a96e59ec324 Tariq Toukan 2020-06-16  137  	}
5229a96e59ec324 Tariq Toukan 2020-06-16  138  
5229a96e59ec324 Tariq Toukan 2020-06-16  139  	flow = mlx5_add_flow_rules(ft->t, spec, &flow_act, &dest, 1);
5229a96e59ec324 Tariq Toukan 2020-06-16  140  
5229a96e59ec324 Tariq Toukan 2020-06-16  141  	if (IS_ERR(flow))
5229a96e59ec324 Tariq Toukan 2020-06-16  142  		netdev_err(priv->netdev, "mlx5_add_flow_rules() failed, flow is %ld\n",
5229a96e59ec324 Tariq Toukan 2020-06-16  143  			   PTR_ERR(flow));
5229a96e59ec324 Tariq Toukan 2020-06-16  144  
5229a96e59ec324 Tariq Toukan 2020-06-16  145  out:
5229a96e59ec324 Tariq Toukan 2020-06-16  146  	kvfree(spec);
5229a96e59ec324 Tariq Toukan 2020-06-16  147  	return flow;
5229a96e59ec324 Tariq Toukan 2020-06-16  148  }
5229a96e59ec324 Tariq Toukan 2020-06-16  149  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#37 lore raw

Re: [PATCH net-next RFC v1 09/10] net/mlx5e: Add NVMEoTCP offload

In reply to Boris Pismenny
reproduce (this is a W=1 build):

Hi Boris,

[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on net-next/master]

url:    https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/990254a46f72010830cdf80ed80ed506f268e759
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308
        git checkout 990254a46f72010830cdf80ed80ed506f268e759
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

In file included from include/linux/bits.h:6,
from include/linux/ratelimit_types.h:5,
from include/linux/ratelimit.h:5,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/rdma/ib_verbs.h:16,
from drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c:33:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]

       7 | #define BIT(nr)   (UL(1) << (nr))
         |                          ^~
   include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
   10680 |  MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
         |                                                                ^~~
   drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c: In function 'mlx5i_grp_sw_update_stats':
>> drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c:143:1: warning: the frame size of 1048 bytes is larger than 1024 bytes [-Wframe-larger-than=]
     143 | }
         | ^

vim +143 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c

48935bbb7ae8bd4 drivers/net/ethernet/mellanox/mlx5/core/ipoib.c       Saeed Mahameed 2017-04-13  117  
fbb66ad5dcbebc1 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Wei Yongjun    2018-09-05  118  static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv)
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  119  {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  120  	struct mlx5e_sw_stats s = { 0 };
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  121  	int i, j;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  122  
694826e366349d5 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Tariq Toukan   2019-07-14  123  	for (i = 0; i < priv->max_nch; i++) {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  124  		struct mlx5e_channel_stats *channel_stats;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  125  		struct mlx5e_rq_stats *rq_stats;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  126  
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  127  		channel_stats = &priv->channel_stats[i];
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  128  		rq_stats = &channel_stats->rq;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  129  
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  130  		s.rx_packets += rq_stats->packets;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  131  		s.rx_bytes   += rq_stats->bytes;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  132  
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  133  		for (j = 0; j < priv->max_opened_tc; j++) {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  134  			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  135  
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  136  			s.tx_packets           += sq_stats->packets;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  137  			s.tx_bytes             += sq_stats->bytes;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  138  			s.tx_queue_dropped     += sq_stats->dropped;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  139  		}
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  140  	}
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  141  
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud    2018-09-02  142  	memcpy(&priv->stats.sw, &s, sizeof(s));

c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 @143 }
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 144

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#38 lore raw

RFC PATCH 10/10 net/mlx5e: NVMEoTCP, data-path for DDP offload

In reply to Boris Pismenny

NVMEoTCP direct data placement constructs an SKB from each CQE, while pointing at NVME buffers.

This enables the offload, as the NVMe-TCP layer will skip the copy when src == dst.

---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |   1 +
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        | 256 ++++++++++++++++++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.h        |  25 ++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  51 +++-
 .../ethernet/mellanox/mlx5/core/en_stats.c    |  12 +
 .../ethernet/mellanox/mlx5/core/en_stats.h    |   8 +
 include/linux/mlx5/device.h                   |  30 +-
 10 files changed, 379 insertions(+), 8 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
index 9dd6b41c2486..89ffb1dae75c 100644
@@ -85,4 +85,4 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o
 					steering/dr_cmd.o steering/dr_fw.o \
 					steering/dr_action.o steering/fs_dr.o
 
-mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
index a8c0fc98b394..47611401a55d 100644
@@ -536,6 +536,7 @@ struct mlx5e_rq;
 typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, struct mlx5_cqe64*);
 typedef struct sk_buff *
 (*mlx5e_fp_skb_from_cqe_mpwrq)(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
+			       struct mlx5_cqe64 *cqe,
 			       u16 cqe_bcnt, u32 head_offset, u32 page_idx);
 typedef struct sk_buff *
 (*mlx5e_fp_skb_from_cqe)(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
index 8e7b877d8a12..9a6fbd1b1c34 100644
@@ -25,6 +25,7 @@ static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, void *data,
 
 struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 						    struct mlx5e_mpw_info *wi,
+						    struct mlx5_cqe64 *cqe,
 						    u16 cqe_bcnt,
 						    u32 head_offset,
 						    u32 page_idx)
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +1 -0
index 7f88ccf67fdd..112c5b3ec165 100644
@@ -11,6 +11,7 @@
 
 struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
 						    struct mlx5e_mpw_info *wi,
+						    struct mlx5_cqe64 *cqe,
 						    u16 cqe_bcnt,
 						    u32 head_offset,
 						    u32 page_idx);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c new file +256 -0

256 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h new file +25 -0
new file mode 100644
index 000000000000..85af1650633c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+// Copyright (c) 2020 Mellanox Technologies.
+
+#ifndef __MLX5E_NVMEOTCP_RXTX_H__
+#define __MLX5E_NVMEOTCP_RXTX_H__
+
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+
+#include <linux/skbuff.h>
+#include "en.h"
+
+struct sk_buff*
+mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
+			     struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear);
+
+int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt);
+#else
+int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { return cqe_bcnt; }
+struct sk_buff*
+mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
+			     struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear) { return skb; }
+
+#endif /* CONFIG_MLX5_EN_NVMEOTCP */
+
+#endif /* __MLX5E_NVMEOTCP_RXTX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +45 -6

143 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +12 -0
index 25d203d64bb2..8fe28694d7cf 100644
@@ -198,6 +198,10 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_drop) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_resync) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_bytes) },
 #endif
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
@@ -331,6 +335,10 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
 		s->rx_nvmeotcp_ddp_setup       += rq_stats->nvmeotcp_ddp_setup;
 		s->rx_nvmeotcp_ddp_setup_fail  += rq_stats->nvmeotcp_ddp_setup_fail;
 		s->rx_nvmeotcp_ddp_teardown    += rq_stats->nvmeotcp_ddp_teardown;
+		s->rx_nvmeotcp_drop            += rq_stats->nvmeotcp_drop;
+		s->rx_nvmeotcp_resync          += rq_stats->nvmeotcp_resync;
+		s->rx_nvmeotcp_offload_packets += rq_stats->nvmeotcp_offload_packets;
+		s->rx_nvmeotcp_offload_bytes   += rq_stats->nvmeotcp_offload_bytes;
 #endif
 		s->ch_events      += ch_stats->events;
 		s->ch_poll        += ch_stats->poll;
@@ -1584,6 +1592,10 @@ static const struct counter_desc rq_stats_desc[] = {
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) },
 	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_drop) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_resync) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_packets) },
+	{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_bytes) },
 #endif
 };
 
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +8 -0
index 5c1c0ad88ff4..be1574e61945 100644
@@ -182,6 +182,10 @@ struct mlx5e_sw_stats {
 	u64 rx_nvmeotcp_ddp_setup;
 	u64 rx_nvmeotcp_ddp_setup_fail;
 	u64 rx_nvmeotcp_ddp_teardown;
+	u64 rx_nvmeotcp_drop;
+	u64 rx_nvmeotcp_resync;
+	u64 rx_nvmeotcp_offload_packets;
+	u64 rx_nvmeotcp_offload_bytes;
 #endif
 	u64 ch_events;
 	u64 ch_poll;
@@ -353,6 +357,10 @@ struct mlx5e_rq_stats {
 	u64 nvmeotcp_ddp_setup;
 	u64 nvmeotcp_ddp_setup_fail;
 	u64 nvmeotcp_ddp_teardown;
+	u64 nvmeotcp_drop;
+	u64 nvmeotcp_resync;
+	u64 nvmeotcp_offload_packets;
+	u64 nvmeotcp_offload_bytes;
 #endif
 };
 
include/linux/mlx5/device.h +29 -1
index afadf4cf6d7a..c1a75f727ade 100644
@@ -779,7 +779,7 @@ struct mlx5_err_cqe {
 
 struct mlx5_cqe64 {
 	u8		tls_outer_l3_tunneled;
-	u8		rsvd0;
+	u8		nvmetcp;
 	__be16		wqe_id;
 	u8		lro_tcppsh_abort_dupack;
 	u8		lro_min_ttl;
@@ -812,6 +812,19 @@ struct mlx5_cqe64 {
 	u8		op_own;
 };
 
+struct mlx5e_cqe128 {
+	__be16		cclen;
+	__be16		hlen;
+	union {
+		__be32		resync_tcp_sn;
+		__be32		ccoff;
+	};
+	__be16		ccid;
+	__be16		rsvd8;
+	u8		rsvd12[52];
+	struct mlx5_cqe64 cqe64;
+};
+
 struct mlx5_mini_cqe8 {
 	union {
 		__be32 rx_hash_result;
@@ -842,6 +855,21 @@ enum {
 
 #define MLX5_MINI_CQE_ARRAY_SIZE 8
 
+static inline bool cqe_is_nvmeotcp_resync(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 6) & 0x1);
+}
+
+static inline bool cqe_is_nvmeotcp_crcvalid(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 5) & 0x1);
+}
+
+static inline bool cqe_is_nvmeotcp_zc(struct mlx5_cqe64 *cqe)
+{
+	return ((cqe->nvmetcp >> 4) & 0x1);
+}
+
 static inline u8 mlx5_get_cqe_format(struct mlx5_cqe64 *cqe)
 {
 	return (cqe->op_own >> 2) & 0x3;
Signature · 1 line
2.24.1
#39 lore raw

Re: [PATCH net-next RFC v1 10/10] net/mlx5e: NVMEoTCP, data-path for DDP offload

reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…

Hi Boris,

[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on net-next/master]

url:    https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1
config: arm64-defconfig (attached as .config)
compiler: aarch64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/26f3c706eccd3eb38acd731a4bfd2583077749bb
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 git checkout 26f3c706eccd3eb38acd731a4bfd2583077749bb
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm64

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

In file included from drivers/net/ethernet/mellanox/mlx5/core/en_rx.c:51:

>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h:18:5: warning: no previous prototype for 'mlx5_nvmeotcp_get_headlen' [-Wmissing-prototypes]
      18 | int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { return cqe_bcnt; }
         |     ^~~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h:20:1: warning: no previous prototype for 'mlx5e_nvmeotcp_handle_rx_skb' [-Wmissing-prototypes]

20 | mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/mlx5_nvmeotcp_get_headlen +18 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h

    11	
    12	struct sk_buff*
    13	mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
    14				     struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear);
    15	
    16	int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt);
    17	#else
  > 18	int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { return cqe_bcnt; }
    19	struct sk_buff*
  > 20	mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
    21				     struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear) { return skb; }
    22	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

maintainer
#40 lore raw
In reply to Boris Pismenny
Quoted 85 lines · Boris Pismenny

This series adds support for nvme-tcp receive offloads which do not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers
2. CRC verification for received PDU

The series implements these as a generic offload infrastructure for storage protocols, which we call TCP Direct Data Placement (TCP_DDP) and TCP DDP CRC, respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC. Future implementations can reuse the same infrastructure for other protcols such as iSCSI.

Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html

Initialization and teardown:
=========================================
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by calling nvme_tcp_offload_socket with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usually with no offload. If it succeeds then nvme_tcp_setup_ddp and nvme_tcp_teardown_ddp may be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of nvme-tcp in any way besides adding the option to call the above mentioned NDO operations.

For the admin queue, nvme-tcp does not initialize the offload. Instead, nvme-tcp calls the driver to configure limits for the controller, such as max_hw_sectors and max_segments; these must be limited to accomodate potential HW resource limits, and to improve performance.

If some error occured, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.

Copy offload works as follows:
=========================================
The nvme-tcp layer calls the NIC drive to map block layer buffers to ccid using nvme_tcp_setup_ddp before sending the read request. When the repsonse is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer; this SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once nvme-tcp attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function (memcpy_to_page):
if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the nvme-tcp layer releases the NIC HW context be calling nvme_tcp_teardown_ddp which asynchronously unmaps the buffers from NIC HW.

As the last change is to a sensative function, we are careful to place it under static_key which is only enabled when this functionality is actually used for nvme-tcp copy offload.

Asynchronous completion:
=========================================
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance criticial, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).

An alternative approach is to move all the functions related to coping from SKBs to the block layer buffers inside the nvme-tcp code - about 200 LOC.

CRC offload works as follows:
=========================================
After offload is initialized, we use the SKB's ddp_crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (ddp_crc != 1), then software must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.

SKB changes:
=========================================
The CRC offload requires an additional bit in the SKB, which is useful for preventing the coalescing of SKB with different crc offload values. This bit is similar in concept to the "decrypted" bit.

Quoted 8 lines

Performance:
=========================================
The expected performance gain from this offload varies with the block size. We perform a CPU cycles breakdown of the copy/CRC operations in nvme-tcp fio random read workloads:
For 4K blocks we see up to 11% improvement for a 100% read fio workload, while for 128K blocks we see upto 52%. If we run nvme-tcp, and skip these operations, then we observe a gain of about 1.1x and 2x respectively.

Nice!

Quoted 33 lines

Resynchronization:
=========================================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the driver, regarding a possible location of a PDU header. Followed by a response from the nvme-tcp driver.

This flow is rare, and it should happen only after packet loss or reordering events that involve nvme-tcp PDU headers.

The patches are organized as follows:
=========================================
Patch 1 the iov_iter change to skip copy if (src == dst)
Patches 2-3 the infrastructure for all TCP DDP
and TCP DDP CRC offloads, respectively.

Patch 4         exposes the get_netdev_for_sock function from TLS
Patch 5         NVMe-TCP changes to call NIC driver on queue init/teardown
Patches 6       NVMe-TCP changes to call NIC driver on IO operation

setup/teardown, and support async completions.
Patches 7 NVMe-TCP changes to support CRC offload on receive.
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patches 8 NVMe-TCP handling of netdev events: stop the offload if
netdev is going down
Patches 9-10 implement support for NVMe-TCP copy and CRC offload in
the mlx5 NIC driver

Testing:
=========================================
This series was tested using fio with various configurations of IO sizes, depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets.

Quoted 5 lines

Future work:
=========================================
A follow-up series will introduce support for transmit side CRC. Then, we will work on adding support for TLS in NVMe-TCP and combining the two offloads.

Boris, Or and Yoray

Thanks for submitting this work. Overall this looks good to me.
The model here is not messy at all which is not trivial when it comes to tcp offloads.

Gave you comments in the patches themselves but overall this looks good!

Would love to see TLS work from you moving forward.