nvme-tcp receive offloads
Subsystems (13)
cover nvme-tcp receive offloads
This series adds support for nvme-tcp receive offloads which do not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers
2. CRC verification for received PDU
The series implements these as a generic offload infrastructure for storage protocols, which we call TCP Direct Data Placement (TCP_DDP) and TCP DDP CRC, respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC. Future implementations can reuse the same infrastructure for other protcols such as iSCSI.
Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html
Initialization and teardown:
=========================================
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by calling nvme_tcp_offload_socket with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usually with no offload. If it succeeds then nvme_tcp_setup_ddp and nvme_tcp_teardown_ddp may be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of nvme-tcp in any way besides adding the option to call the above mentioned NDO operations.
For the admin queue, nvme-tcp does not initialize the offload. Instead, nvme-tcp calls the driver to configure limits for the controller, such as max_hw_sectors and max_segments; these must be limited to accomodate potential HW resource limits, and to improve performance.
If some error occured, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.
Copy offload works as follows:
=========================================
The nvme-tcp layer calls the NIC drive to map block layer buffers to ccid using nvme_tcp_setup_ddp before sending the read request. When the repsonse is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer; this SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once nvme-tcp attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function (memcpy_to_page):
if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the nvme-tcp layer releases the NIC HW context be calling nvme_tcp_teardown_ddp which asynchronously unmaps the buffers from NIC HW.
As the last change is to a sensative function, we are careful to place it under static_key which is only enabled when this functionality is actually used for nvme-tcp copy offload.
Asynchronous completion:
=========================================
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance criticial, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).
An alternative approach is to move all the functions related to coping from SKBs to the block layer buffers inside the nvme-tcp code - about 200 LOC.
CRC offload works as follows:
=========================================
After offload is initialized, we use the SKB's ddp_crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (ddp_crc != 1), then software must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.
SKB changes:
=========================================
The CRC offload requires an additional bit in the SKB, which is useful for preventing the coalescing of SKB with different crc offload values. This bit is similar in concept to the "decrypted" bit.
Performance:
=========================================
The expected performance gain from this offload varies with the block size. We perform a CPU cycles breakdown of the copy/CRC operations in nvme-tcp fio random read workloads:
For 4K blocks we see up to 11% improvement for a 100% read fio workload, while for 128K blocks we see upto 52%. If we run nvme-tcp, and skip these operations, then we observe a gain of about 1.1x and 2x respectively.
Resynchronization:
=========================================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the driver, regarding a possible location of a PDU header. Followed by a response from the nvme-tcp driver.
This flow is rare, and it should happen only after packet loss or reordering events that involve nvme-tcp PDU headers.
The patches are organized as follows:
=========================================
Patch 1 the iov_iter change to skip copy if (src == dst)
Patches 2-3 the infrastructure for all TCP DDP
and TCP DDP CRC offloads, respectively.
Patch 4 exposes the get_netdev_for_sock function from TLS Patch 5 NVMe-TCP changes to call NIC driver on queue init/teardown Patches 6 NVMe-TCP changes to call NIC driver on IO operation
setup/teardown, and support async completions.
Patches 7 NVMe-TCP changes to support CRC offload on receive.
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patches 8 NVMe-TCP handling of netdev events: stop the offload if
netdev is going down
Patches 9-10 implement support for NVMe-TCP copy and CRC offload in
the mlx5 NIC driver
Testing:
=========================================
This series was tested using fio with various configurations of IO sizes, depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets.
Future work:
=========================================
A follow-up series will introduce support for transmit side CRC. Then, we will work on adding support for TLS in NVMe-TCP and combining the two offloads.
Boris Pismenny (8):
iov_iter: Skip copy in memcpy_to_page if src==dst
net: Introduce direct data placement tcp offload
net: Introduce crc offload for tcp ddp ulp net/tls: expose get_netdev_for_sock
nvme-tcp: Add DDP offload control path
nvme-tcp: Add DDP data-path
net/mlx5e: Add NVMEoTCP offload net/mlx5e: NVMEoTCP, data-path for DDP offload
Or Gerlitz (1):
nvme-tcp: Deal with netdevice DOWN events
Yoray Zack (1):
nvme-tcp : Recalculate crc in the end of the capsule
.../net/ethernet/mellanox/mlx5/core/Kconfig | 11 + .../net/ethernet/mellanox/mlx5/core/Makefile | 2 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 12 +- .../ethernet/mellanox/mlx5/core/en/params.h | 4 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 13 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 1 + .../mellanox/mlx5/core/en_accel/en_accel.h | 9 +- .../mellanox/mlx5/core/en_accel/fs_tcp.c | 10 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 894 ++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 116 +++ .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 256 +++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.h | 25 + .../mlx5/core/en_accel/nvmeotcp_utils.h | 79 ++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 73 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 76 +- .../ethernet/mellanox/mlx5/core/en_stats.c | 38 + .../ethernet/mellanox/mlx5/core/en_stats.h | 24 + .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 6 + drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + drivers/nvme/host/tcp.c | 410 +++++++- include/linux/mlx5/device.h | 38 +- include/linux/mlx5/mlx5_ifc.h | 121 ++- include/linux/mlx5/qp.h | 1 + include/linux/netdev_features.h | 4 + include/linux/netdevice.h | 5 + include/linux/nvme-tcp.h | 2 + include/linux/skbuff.h | 4 + include/linux/uio.h | 2 + include/net/inet_connection_sock.h | 4 + include/net/sock.h | 17 + include/net/tcp_ddp.h | 90 ++ lib/iov_iter.c | 11 +- net/Kconfig | 17 + net/core/skbuff.c | 9 +- net/ethtool/common.c | 2 + net/ipv4/tcp_input.c | 7 + net/ipv4/tcp_ipv4.c | 3 + net/ipv4/tcp_offload.c | 3 + net/tls/tls_device.c | 20 +- 40 files changed, 2375 insertions(+), 51 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h create mode 100644 include/net/tcp_ddp.h
base-commit: 300fd579b2e8608586b002207e906ac95c74b911
prerequisite-patch-id: b3079fa1f4c0e3d6d4a92f59e70981e8a28f3b83
Signature · 1 line
2.24.1
RFC PATCH 1/10 iov_iter: Skip copy in memcpy_to_page if src==dst
When using direct data placement the NIC writes some of the payload directly to the destination buffer, and constructs the SKB such that it points to this data. As a result, the skb_copy datagram_iter call will attempt to copy data when it is not necessary.
This patch adds a check to avoid this copy, and a static_key to enabled it when TCP direct data placement is possible.
--- include/linux/uio.h | 2 ++ lib/iov_iter.c | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
include/linux/uio.h +2 -0
@@ -284,4 +284,6 @@ int iov_iter_for_each_range(struct iov_iter *i, size_t bytes,int (*f)(struct kvec *vec,void *context), void *context); +extern struct static_key_false skip_copy_enabled; +#endif
lib/iov_iter.c +10 -1
@@ -13,6 +13,9 @@#define PIPE_PARANOIA /* for now */ +DEFINE_STATIC_KEY_FALSE (skip_copy_enabled); +EXPORT_SYMBOL_GPL (skip_copy_enabled); +#define iterate_iovec (i, n, __v, __p, skip,STEP ) { \ size_t left; \ size_t wanted = n; \
@@ -470,7 +473,13 @@ static void memcpy_from_page(char *to, struct page *page, size_t offset, size_tstatic void memcpy_to_page (struct page *page,size_t offset,const char *from,size_t len) {char *to =kmap_atomic (page); -memcpy (to + offset, from, len); + +if (static_branch_unlikely (&skip_copy_enabled)) { +if (to + offset != from) +memcpy (to + offset, from, len); + }else { +memcpy (to + offset, from, len); + }kunmap_atomic (to); }
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 01/10] iov_iter: Skip copy in memcpy_to_page if src==dst
You probably want Al to have a look at this..
RFC PATCH 2/10 net: Introduce direct data placement tcp offload
In reply to Boris PismennyThis commit introduces direct data placement offload for TCP. This capability is accompanied by new net_device operations that configure
hardware contexts. There is a context per socket, and a context per DDP opreation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.
This, the SKB represents the data on the wire, while it is pointing to data that is already placed in the destination buffer. As a result, data from page frags should not be copied out to the linear part.
As SKBs that use DDP are already very memory efficient, we modify skb_condence to avoid copying data from fragments to the linear part of SKBs that belong to a socket that uses DDP offload.
A follow-up patch will use this interface for DDP in NVMe-TCP.
--- include/linux/netdev_features.h | 2 + include/linux/netdevice.h | 5 ++ include/net/inet_connection_sock.h | 4 ++ include/net/tcp_ddp.h | 90 ++++++++++++++++++++++++++++++ net/Kconfig | 9 +++ net/core/skbuff.c | 9 ++- net/ethtool/common.c | 1 + 7 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 include/net/tcp_ddp.h
include/linux/netdev_features.h +2 -0
@@ -84,6 +84,7 @@ enum {NETIF_F_GRO_FRAGLIST_BIT ,/* Fraglist GRO */ NETIF_F_HW_MACSEC_BIT ,/* Offload MACsec operations */ + NETIF_F_HW_TCP_DDP_BIT, /* TCP direct data placement offload *//* * Add your fresh new feature above and remember to update
@@ -157,6 +158,7 @@ enum {#define NETIF_F_GRO_FRAGLIST __NETIF_F(GRO_FRAGLIST)#define NETIF_F_GSO_FRAGLIST __NETIF_F(GSO_FRAGLIST)#define NETIF_F_HW_MACSEC __NETIF_F(HW_MACSEC) +#define NETIF_F_HW_TCP_DDP __NETIF_F(HW_TCP_DDP)/* Finds the next feature with the highest number of the range of start till 0. */
include/linux/netdevice.h +5 -0
@@ -935,6 +935,7 @@ struct dev_ifalias {struct devlink ;struct tlsdev_ops ; +struct tcp_ddp_dev_ops ;struct netdev_name_node {struct hlist_node hlist;
@@ -1922,6 +1923,10 @@ struct net_device {const struct tlsdev_ops *tlsdev_ops;#endif +#ifdef CONFIG_TCP_DDP +const struct tcp_ddp_dev_ops *tcp_ddp_ops; +#endif +const struct header_ops *header_ops;unsigned int flags;
include/net/inet_connection_sock.h +4 -0
@@ -66,6 +66,8 @@ struct inet_connection_sock_af_ops {* @icsk_ulp_ops Pluggable ULP control hook * @icsk_ulp_data ULP private data * @icsk_clean_acked Clean acked data hook +* @icsk_ulp_ddp_ops Pluggable ULP direct data placement control hook +* @icsk_ulp_ddp_data ULP direct data placement private data * @icsk_listen_portaddr_node hash to the portaddr listener hashtable * @icsk_ca_state: Congestion control state * @icsk_retransmits: Number of unrecovered [RTO] timeouts
@@ -94,6 +96,8 @@ struct inet_connection_sock {const struct tcp_ulp_ops *icsk_ulp_ops;void __rcu *icsk_ulp_data;void (*icsk_clean_acked)(struct sock *sk,u32 acked_seq); +const struct tcp_ddp_ulp_ops *icsk_ulp_ddp_ops; +void __rcu *icsk_ulp_ddp_data;struct hlist_node icsk_listen_portaddr_node;unsigned int (*icsk_sync_mss)(struct sock *sk,u32 pmtu); __u8 icsk_ca_state:5,
include/net/tcp_ddp.h new file +90 -0
90 lines not on this page · raw message
net/Kconfig +9 -0
@@ -460,6 +460,15 @@ config ETHTOOL_NETLINK netlink. It provides better extensibility and some new features, e.g. notification messages. +config TCP_DDP + bool "TCP direct data placement offload" + default n + help + Direct Data Placement (DDP) offload for TCP enables ULP, such as + NVMe-TCP/iSCSI, to request the NIC to place TCP payload data + of a command response directly into kernel pages. + + endif # if NET # Used by archs to tell that they support BPF JIT compiler plus which flavour.
net/core/skbuff.c +8 -1
@@ -69,6 +69,7 @@#include <net/xfrm.h> #include <net/mpls.h> #include <net/mptcp.h> +#include <net/tcp_ddp.h> #include <linux/uaccess.h> #include <trace/events/skb.h>
@@ -6059,9 +6060,15 @@ EXPORT_SYMBOL(pskb_extract);*/ void skb_condense (struct sk_buff *skb) { +bool is_ddp = false; + +#ifdef CONFIG_TCP_DDP + is_ddp = skb->sk &&inet_csk (skb->sk) && +inet_csk (skb->sk)->icsk_ulp_ddp_data; +#endif if (skb->data_len) {if (skb->data_len > skb->end - skb->tail || -skb_cloned (skb)) +skb_cloned (skb) || is_ddp)return ;/* Nice, we can free page frag(s) right now */
net/ethtool/common.c +1 -0
@@ -68,6 +68,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_HW_TLS_RX_BIT] = "tls-hw-rx-offload", [NETIF_F_GRO_FRAGLIST_BIT] = "rx-gro-list", [NETIF_F_HW_MACSEC_BIT] = "macsec-hw-offload", + [NETIF_F_HW_TCP_DDP_BIT] = "tcp-ddp-offload", }; const char
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 02/10] net: Introduce direct data placement tcp offload
include/net/tcp_ddp.h line 10
Quoted 9 lines
+ *
tcp_ddp.h
+ * Author: Boris Pismenny <[email protected]>
+ * Copyright (C) 2020 Mellanox Technologies.
+ */include/net/tcp_ddp.h new file +4
-0+#ifndef _TCP_DDP_H +#define _TCP_DDP_H + +#include <linux/blkdev.h>
Why is blkdev.h needed?
Quoted 6 lines
include/net/tcp_ddp.h new file +6
-0+#include <linux/netdevice.h> +#include <net/inet_connection_sock.h> +#include <net/sock.h> + +/* limits returned by the offload driver, zero means don't care */ +struct tcp_ddp_limits {
include/net/tcp_ddp.h line 26
Quoted 10 lines
include/net/tcp_ddp.h new file +10
-0+int max_ddp_sgl_len; +}; + +enum tcp_ddp_type { +TCP_DDP_NVME =1 , +}; + +struct tcp_ddp_config { +enum tcp_ddp_type type; +unsigned char buf[];
A little kdoc may help here as its not exactly clear what is buf used for (at this point at least)...
include/net/tcp_ddp.h line 29
Quoted 3 lines
include/net/tcp_ddp.h new file +3
-0+}; + +struct nvme_tcp_config {
struct nvme_tcp_ddp_config
include/net/tcp_ddp.h line 39
Quoted 10 lines
include/net/tcp_ddp.h new file +10
-0+struct tcp_ddp_config cfg; + +u16 pfv; +u8 cpda; +u8 dgst; +int queue_size; +int queue_id; +int io_cpu; +}; +
Other than that this looks good to me.
Re: [PATCH net-next RFC v1 02/10] net: Introduce direct data placement tcp offload
Quoted 9 lines · Sagi Grimberg
include/net/tcp_ddp.h line 10
Quoted 8 lines
+ *
tcp_ddp.h
+ * Author: Boris Pismenny <[email protected]>
+ * Copyright (C) 2020 Mellanox Technologies.
+ */include/net/tcp_ddp.h new file +4
-0+#ifndef _TCP_DDP_H +#define _TCP_DDP_H + +#include <linux/blkdev.h> Why is
blkdev.hneeded?
That's a lefotover from a previous iteration over this code. I'll remove it for the next patch.
Quoted 11 lines
Quoted 11 lines
include/net/tcp_ddp.h new file +10
-0+#include <linux/netdevice.h> +#include <net/inet_connection_sock.h> +#include <net/sock.h> + +/* limits returned by the offload driver, zero means don't care */ +struct tcp_ddp_limits { +int max_ddp_sgl_len; +}; + +enum tcp_ddp_type {
Quoted 8 lines
include/net/tcp_ddp.h line 26
Quoted 6 lines
include/net/tcp_ddp.h new file +6
-0+ TCP_DDP_NVME = 1, +}; + +struct tcp_ddp_config { +enum tcp_ddp_type type; +unsigned char buf[];A little kdoc may help here as its not exactly clear what is buf used for (at this point at least)...
Will add.
Quoted 4 lines
include/net/tcp_ddp.h line 29
Quoted 3 lines
include/net/tcp_ddp.h new file +3
-0+}; + +struct nvme_tcp_config {struct
nvme_tcp_ddp_config
Sure.
Quoted 11 lines
include/net/tcp_ddp.h line 39
Quoted 10 lines
include/net/tcp_ddp.h new file +10
-0+struct tcp_ddp_config cfg; + +u16 pfv; +u8 cpda; +u8 dgst; +int queue_size; +int queue_id; +int io_cpu; +}; +Other than that this looks good to me.
Thanks Sagi!
RFC PATCH 3/10 net: Introduce crc offload for tcp ddp ulp
In reply to Boris PismennyThis commit itroduces support for CRC offload to direct data placement ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.
On the receive side, CRC offload requires a new SKB bit that indicates that no CRC error was encountered while processing this packet. If all packets of a ULP message have this bit set, then the CRC verification for the message can be skipped, as hardware already checked it.
The following patches will set and use this bit to perform NVME-TCP CRC offload.
A subsequent series, will add NVMe-TCP transmit side CRC support.
--- include/linux/netdev_features.h | 2 ++ include/linux/skbuff.h | 4 ++++ net/Kconfig | 8 ++++++++ net/ethtool/common.c | 1 + net/ipv4/tcp_input.c | 7 +++++++ net/ipv4/tcp_ipv4.c | 3 +++ net/ipv4/tcp_offload.c | 3 +++ 7 files changed, 28 insertions(+)
include/linux/netdev_features.h +2 -0
@@ -85,6 +85,7 @@ enum {NETIF_F_HW_MACSEC_BIT ,/* Offload MACsec operations */ NETIF_F_HW_TCP_DDP_BIT ,/* TCP direct data placement offload */ + NETIF_F_HW_TCP_DDP_CRC_BIT, /* TCP DDP CRC offload *//* * Add your fresh new feature above and remember to update
@@ -159,6 +160,7 @@ enum {#define NETIF_F_GSO_FRAGLIST __NETIF_F(GSO_FRAGLIST)#define NETIF_F_HW_MACSEC __NETIF_F(HW_MACSEC)#define NETIF_F_HW_TCP_DDP __NETIF_F(HW_TCP_DDP) +#define NETIF_F_HW_TCP_DDP_CRC __NETIF_F(HW_TCP_DDP_CRC)/* Finds the next feature with the highest number of the range of start till 0. */
include/linux/skbuff.h +4 -0
@@ -858,6 +858,10 @@ struct sk_buff {#ifdef CONFIG_TLS_DEVICE __u8 decrypted:1;#endif +#ifdef CONFIG_TCP_DDP_CRC + __u8 ddp_crc:1; +#endif +#ifdef CONFIG_NET_SCHED __u16 tc_index;/* traffic control index */
net/Kconfig +8 -0
@@ -468,6 +468,14 @@ config TCP_DDP NVMe-TCP/iSCSI, to request the NIC to place TCP payload data of a command response directly into kernel pages. +config TCP_DDP_CRC + bool "TCP direct data placement CRC offload" + default n + help + Direct Data Placement (DDP) CRC32C offload for TCP enables ULP, such as + NVMe-TCP/iSCSI, to request the NIC to calculate/verify the data digest + of commands as they go through the NIC. Thus avoiding the costly + per-byte overhead. endif # if NET
net/ethtool/common.c +1 -0
@@ -69,6 +69,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_GRO_FRAGLIST_BIT] = "rx-gro-list", [NETIF_F_HW_MACSEC_BIT] = "macsec-hw-offload", [NETIF_F_HW_TCP_DDP_BIT] = "tcp-ddp-offload", + [NETIF_F_HW_TCP_DDP_CRC_BIT] = "tcp-ddp-crc-offload", }; const char
net/ipv4/tcp_input.c +7 -0
@@ -5099,6 +5099,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,memcpy (nskb->cb, skb->cb,sizeof (skb->cb));#ifdef CONFIG_TLS_DEVICE nskb->decrypted = skb->decrypted; +#endif +#ifdef CONFIG_TCP_DDP_CRC + nskb->ddp_crc = skb->ddp_crc;#endif TCP_SKB_CB (nskb)->seq =TCP_SKB_CB (nskb)->end_seq = start;if (list)
@@ -5132,6 +5135,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,#ifdef CONFIG_TLS_DEVICE if (skb->decrypted != nskb->decrypted) goto end; +#endif +#ifdef CONFIG_TCP_DDP_CRC +if (skb->ddp_crc != nskb->ddp_crc) + goto end;#endif } }
net/ipv4/tcp_ipv4.c +3 -0
@@ -1791,6 +1791,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_ECE | TCPHDR_CWR)) ||#ifdef CONFIG_TLS_DEVICE tail->decrypted != skb->decrypted || +#endif +#ifdef CONFIG_TCP_DDP_CRC + tail->ddp_crc != skb->ddp_crc || #endif thtail->doff != th->doff || memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)))
net/ipv4/tcp_offload.c +3 -0
@@ -262,6 +262,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)#ifdef CONFIG_TLS_DEVICE flush |= p->decrypted ^ skb->decrypted;#endif +#ifdef CONFIG_TCP_DDP_CRC + flush |= p->ddp_crc ^ skb->ddp_crc; +#endif if (flush ||skb_gro_receive (p, skb)) { mss =1 ;
Signature · 1 line
2.24.1
Re: [PATCH net-next RFC v1 03/10] net: Introduce crc offload for tcp ddp ulp
Quoted 2 lines
This commit itroduces support for CRC offload to direct data placement
introduces
Quoted 3 lines
ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.
From the API it not clear that the offload engine does crc32c, do you see this extended to other crc types in the future?
Other than this the patch looks good to me,
Re: [PATCH net-next RFC v1 03/10] net: Introduce crc offload for tcp ddp ulp
Quoted 7 lines · Sagi Grimberg
Quoted 1 line
This commit itroduces support for CRC offload to direct data placement
introduces
Quoted 3 lines
ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently.
From the API it not clear that the offload engine does crc32c, do you
see this extended to other crc types in the future?
Yes, it is somewhat implicit, and it depends on the tcp ddp configuration. At the moment we only do nvme-tcp, and that has only CRC32C. If in the future, there would be other CRC variants, or data digest algorithms, then the code could be easily extended.
In general, any data digest over TCP can leverage this code and SKB member. Not only other CRC types can benefit from it, but even more complex data digest algorithms like SHA can use this. In essence this bit is similar to the TLS skb->decrypted bit. In TLS, skb->decrypted also indicates that the authentication has no error, exactly like ddp_crc indicates that there is no CRC32C error. The only reason we didn't use the same bit for both is that these two protocol offloads can be combined and that will benefit from two independent bits.
Quoted 3 lines
Other than this the patch looks good to me,
Reviewed-by Sagi Grimberg <[email protected]>
RFC PATCH 4/10 net/tls: expose get_netdev_for_sock
In reply to Boris Pismennyget_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket.
Later patches will use this for nvme-tcp DDP and DDP CRC offloads.
--- include/net/sock.h | 17 +++++++++++++++++ net/tls/tls_device.c | 20 ++------------------ 2 files changed, 19 insertions(+), 18 deletions(-)
include/net/sock.h +17 -0
@@ -2703,4 +2703,21 @@ void sock_set_sndtimeo(struct sock *sk, s64 secs);int sock_bind_add (struct sock *sk,struct sockaddr *addr,int addr_len); +/* Assume that the socket is already connected */ +static inline struct net_device *get_netdev_for_sock (struct sock *sk,bool hold) +{ +struct dst_entry *dst =sk_dst_get (sk); +struct net_device *netdev =NULL ; + +if (likely (dst)) { + netdev = dst->dev; +if (hold) +dev_hold (netdev); + } + +dst_release (dst); + +return netdev; +} +#endif /* _SOCK_H */
net/tls/tls_device.c +2 -18
@@ -106,22 +106,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx)spin_unlock_irqrestore (&tls_device_lock, flags); } -/* We assume that the socket is already connected */ -static struct net_device *get_netdev_for_sock (struct sock *sk) -{ -struct dst_entry *dst =sk_dst_get (sk); -struct net_device *netdev =NULL ; - -if (likely (dst)) { - netdev = dst->dev; -dev_hold (netdev); - } - -dst_release (dst); - -return netdev; -} -static void destroy_record (struct tls_record_info *record) {int i;
@@ -1086,7 +1070,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)if (skb)TCP_SKB_CB (skb)->eor =1 ; - netdev =get_netdev_for_sock (sk); + netdev =get_netdev_for_sock (sk, true);if (!netdev) {pr_err_ratelimited ("%s: netdev not found\n" , __func__); rc = -EINVAL ;
@@ -1162,7 +1146,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)if (ctx->crypto_recv.info.version !=TLS_1_2_VERSION )return -EOPNOTSUPP ; - netdev =get_netdev_for_sock (sk); + netdev =get_netdev_for_sock (sk, true);if (!netdev) {pr_err_ratelimited ("%s: netdev not found\n" , __func__);return -EINVAL ;
Signature · 1 line
2.24.1
Re: [PATCH net-next RFC v1 04/10] net/tls: expose get_netdev_for_sock
RFC PATCH 5/10 nvme-tcp: Add DDP offload control path
In reply to Boris PismennyThis commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via tcp_ddp_limits.
A follow-up patch introduces the data-path changes required for this offload.
--- drivers/nvme/host/tcp.c | 188 +++++++++++++++++++++++++++++++++++++++ include/linux/nvme-tcp.h | 2 + 2 files changed, 190 insertions(+)
drivers/nvme/host/tcp.c +188 -0
@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3, }; enum nvme_tcp_recv_state {
@@ -110,6 +111,8 @@ struct nvme_tcp_queue {void (*state_change)(struct sock *);void (*data_ready)(struct sock *);void (*write_space)(struct sock *); + +atomic64_t resync_req; };struct nvme_tcp_ctrl {
@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; };static LIST_HEAD (nvme_tcp_ctrl_list);
@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +#ifdef CONFIG_TCP_DDP + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops__read_mostly = { + .resync_request = nvme_tcp_resync_request, +}; + +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config; +int ret; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ; + } + +if (!(netdev->features &NETIF_F_HW_TCP_DDP )) { +dev_put (netdev); +return -EINVAL ; + } + + ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add (netdev, + queue->sock->sk, + ddp_config); +if (!ret) +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; +else +dev_put (netdev); +return ret; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return ; + } + + netdev->tcp_ddp_ops->tcp_ddp_sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ; +dev_put (netdev);/* put the queue_init get_netdev_for_sock() */ +} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +int ret =0 ; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ; + } + +if (netdev->features &NETIF_F_HW_TCP_DDP && + netdev->tcp_ddp_ops && + netdev->tcp_ddp_ops->tcp_ddp_limits) + ret = netdev->tcp_ddp_ops->tcp_ddp_limits (netdev, limits); +else + ret = -EOPNOTSUPP ; + +if (!ret) { + queue->ctrl->offloading_netdev = netdev; +pr_info ("%s netdev %s offload limits: max_ddp_sgl_len %d\n" , + __func__, netdev->name, limits->max_ddp_sgl_len); + queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); + }else { + queue->ctrl->offloading_netdev =NULL ; + } + +dev_put (netdev); + +return ret; +} + +static +void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 resync_val; +u32 resync_seq; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return ; + } + + resync_val =atomic64_read (&queue->resync_req); +if ((resync_val &TCP_DDP_RESYNC_REQ ) ==0 ) +return ; + + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ; + +if (atomic64_cmpxchg (&queue->resync_req, resync_val, (resync_val -1 ))) + netdev->tcp_ddp_ops->tcp_ddp_resync (netdev, queue->sock->sk, pdu_seq); +} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +atomic64_set (&queue->resync_req, + (((uint64_t )seq <<32 ) | flags)); + +return true; +} + +#else + +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +return -EINVAL ; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +return -EINVAL ; +} + +static +void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +return false; +} + +#endif +static void nvme_tcp_init_iter (struct nvme_tcp_request *req,unsigned int dir) {
@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; +u64 pdu_seq =TCP_SKB_CB (skb)->seq + *offset - queue->pdu_offset; + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_resync_response (queue, pdu_seq); + ret =skb_copy_bits (skb, *offset, &pdu[queue->pdu_offset], rcv_len);if (unlikely (ret))
@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[qid]; +struct nvme_tcp_config config; +struct tcp_ddp_limits limits;int ret, rcv_pdu_size; queue->ctrl = ctrl;
@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,#endif write_unlock_bh (&queue->sock->sk->sk_callback_lock); +if (nvme_tcp_queue_id (queue) !=0 ) { + config.cfg.type =TCP_DDP_NVME ; + config.pfv =NVME_TCP_PFV_1_0 ; + config.cpda =0 ; + config.dgst = queue->hdr_digest ? +NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.dgst |= queue->data_digest ? +NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.queue_size = queue->queue_size; + config.queue_id =nvme_tcp_queue_id (queue); + config.io_cpu = queue->io_cpu; + + ret =nvme_tcp_offload_socket (queue, &config); +if (!ret) +set_bit (NVME_TCP_Q_OFFLOADS , &queue->flags); + }else { + ret =nvme_tcp_offload_limits (queue, &limits); + } +/* offload is opportunistic - failure is non-critical */ +return 0 ; err_init_connect:
@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_calls (queue);cancel_work_sync (&queue->io_work); + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_unoffload_socket (queue); }static void nvme_tcp_stop_queue (struct nvme_ctrl *nctrl,int qid)
include/linux/nvme-tcp.h +2 -0
@@ -8,6 +8,8 @@#define _LINUX_NVME_TCP_H#include <linux/nvme.h> +#include <net/sock.h> +#include <net/tcp_ddp.h> #define NVME_TCP_DISC_PORT 8009#define NVME_TCP_ADMIN_CCSZ SZ_8K
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path
Quoted 75 lines · Boris Pismenny
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
tcp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 188 +++++++++++++++++++++++++++++++++++++++ include/linux/nvme-tcp.h | 2 + 2 files changed, 190 insertions(+)drivers/nvme/host/tcp.c +11
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3, }; enum nvme_tcp_recv_state {@@ -110,6 +111,8 @@ struct nvme_tcp_queue {void (*state_change)(struct sock *);void (*data_ready)(struct sock *);void (*write_space)(struct sock *); + +atomic64_t resync_req; };struct nvme_tcp_ctrl {@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; };static LIST_HEAD (nvme_tcp_ctrl_list);@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +#ifdef CONFIG_TCP_DDP + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops__read_mostly = { + .resync_request = nvme_tcp_resync_request, +};
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 246
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config; +int ret; + +if (unlikely (!netdev)) {
Let's remove unlikely from non datapath routines, its slightly confusing.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 247
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+pr_info_ratelimited ("%s: netdev not found\n" , __func__);
dev_info_ratelimited with queue->ctrl->ctrl.device ? Also, lets remove __func__. This usually is not very helpful.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 253
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+return -EINVAL ; + } + +if (!(netdev->features &NETIF_F_HW_TCP_DDP )) { +dev_put (netdev); +return -EINVAL ;
EINVAL or ENODEV?
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ } + + ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add (netdev, + queue->sock->sk, + ddp_config); +if (!ret) +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; +else +dev_put (netdev);
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 272
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+return ret; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__);
Same here.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 278
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+return ; + } + + netdev->tcp_ddp_ops->tcp_ddp_sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ;
Just a general question, why is this needed?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 291
Quoted 13 lines
drivers/nvme/host/tcp.c +13
-0+dev_put (netdev);/* put the queue_init get_netdev_for_sock() */ +} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +int ret =0 ; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ;
Same here
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 304
Quoted 13 lines
drivers/nvme/host/tcp.c +13
-0+ } + +if (netdev->features &NETIF_F_HW_TCP_DDP && + netdev->tcp_ddp_ops && + netdev->tcp_ddp_ops->tcp_ddp_limits) + ret = netdev->tcp_ddp_ops->tcp_ddp_limits (netdev, limits); +else + ret = -EOPNOTSUPP ; + +if (!ret) { + queue->ctrl->offloading_netdev = netdev; +pr_info ("%s netdev %s offload limits: max_ddp_sgl_len %d\n" , + __func__, netdev->name, limits->max_ddp_sgl_len);
dev_info, and given that it per-queue, please make it dev_dbg.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 309
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); + }else { + queue->ctrl->offloading_netdev =NULL ;
Maybe nullify in the controller setup intead?
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ } + +dev_put (netdev); + +return ret; +} + +static
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 327
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 resync_val; +u32 resync_seq; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return ;
What happens now, fallback to SW? Maybe dev_warn then.. will the SW keep seeing these responses after one failed?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 336
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ } + + resync_val =atomic64_read (&queue->resync_req); +if ((resync_val &TCP_DDP_RESYNC_REQ ) ==0 ) +return ; + + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ;
I think it will be better to pass the skb to this func and keep the pdu_seq contained locally.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 339
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ +if (atomic64_cmpxchg (&queue->resync_req, resync_val, (resync_val -1 ))) + netdev->tcp_ddp_ops->tcp_ddp_resync (netdev, queue->sock->sk, pdu_seq);
A small comment on this manipulation may help the reader.
Quoted 48 lines
drivers/nvme/host/tcp.c +44
-0+} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +atomic64_set (&queue->resync_req, + (((uint64_t )seq <<32 ) | flags)); + +return true; +} + +#else + +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +return -EINVAL ; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +return -EINVAL ; +} + +static +void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +return false; +} + +#endif +static void nvme_tcp_init_iter (struct nvme_tcp_request *req,unsigned int dir) {
drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 792
Quoted 7 lines
drivers/nvme/host/tcp.c +4
-0@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; +u64 pdu_seq =TCP_SKB_CB (skb)->seq + *offset - queue->pdu_offset; + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_resync_response (queue, pdu_seq);
Here, just pass in (queue, skb)
Quoted 14 lines
drivers/nvme/host/tcp.c +3
-0+ ret =skb_copy_bits (skb, *offset, &pdu[queue->pdu_offset], rcv_len);if (unlikely (ret))@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[qid]; +struct nvme_tcp_config config; +struct tcp_ddp_limits limits;int ret, rcv_pdu_size; queue->ctrl = ctrl;
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1655
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,#endif write_unlock_bh (&queue->sock->sk->sk_callback_lock); +if (nvme_tcp_queue_id (queue) !=0 ) {
if (!nvme_tcp_admin_queue(queue)) {drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1665
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ config.cfg.type =TCP_DDP_NVME ; + config.pfv =NVME_TCP_PFV_1_0 ; + config.cpda =0 ; + config.dgst = queue->hdr_digest ? +NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.dgst |= queue->data_digest ? +NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.queue_size = queue->queue_size; + config.queue_id =nvme_tcp_queue_id (queue); + config.io_cpu = queue->io_cpu;
Can the config initialization move to nvme_tcp_offload_socket?
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1672
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ + ret =nvme_tcp_offload_socket (queue, &config); +if (!ret) +set_bit (NVME_TCP_Q_OFFLOADS , &queue->flags); + }else { + ret =nvme_tcp_offload_limits (queue, &limits); + }
I'm thinking that instead of this conditional, we want to place nvme nvme_tcp_alloc_admin_queue in nvme_tcp_alloc_admin_queue, and also move nvme_tcp_alloc_admin_queue to __nvme_tcp_alloc_io_queues loop.
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1673
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+/* offload is opportunistic - failure is non-critical */
Than make it void...
drivers/nvme/host/tcp.c static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) · line 1709
Quoted 11 lines
drivers/nvme/host/tcp.c +4
-0+return 0 ; err_init_connect:@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_calls (queue);cancel_work_sync (&queue->io_work); + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_unoffload_socket (queue);
Why not in nvme_tcp_free_queue, symmetric to the alloc?
include/linux/nvme-tcp.h line 12
Quoted 13 lines
drivers/nvme/host/tcp.c +0
-0}static void nvme_tcp_stop_queue (struct nvme_ctrl *nctrl,int qid)include/linux/nvme-tcp.h +2
-0@@ -8,6 +8,8 @@#define _LINUX_NVME_TCP_H#include <linux/nvme.h> +#include <net/sock.h> +#include <net/tcp_ddp.h>
Why is this needed? I think we want to place this in tcp.c no?
Re: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path
Quoted 93 lines · Sagi Grimberg
Quoted 75 lines · Boris Pismenny
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake
using the tcp_ddp_sk_add/del NDOs.Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW,
and resume offload when all is successful.Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
tcp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 188 +++++++++++++++++++++++++++++++++++++++ include/linux/nvme-tcp.h | 2 + 2 files changed, 190 insertions(+)drivers/nvme/host/tcp.c +11
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3, }; enum nvme_tcp_recv_state {@@ -110,6 +111,8 @@ struct nvme_tcp_queue {void (*state_change)(struct sock *);void (*data_ready)(struct sock *);void (*write_space)(struct sock *); + +atomic64_t resync_req; };struct nvme_tcp_ctrl {@@ -129,6 +132,8 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; };static LIST_HEAD (nvme_tcp_ctrl_list);@@ -223,6 +228,159 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +#ifdef CONFIG_TCP_DDP + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops__read_mostly = { + .resync_request = nvme_tcp_resync_request, +};drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 246
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +struct tcp_ddp_config *ddp_config = (struct tcp_ddp_config *)config; +int ret; + +if (unlikely (!netdev)) {Let's remove unlikely from non datapath routines, its slightly confusing.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 247
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+pr_info_ratelimited ("%s: netdev not found\n" , __func__);
dev_info_ratelimitedwithqueue->ctrl->ctrl.device? Also, lets remove__func__. This usually is not very helpful.
Quoted 7 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 253
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+return -EINVAL ; + } + +if (!(netdev->features &NETIF_F_HW_TCP_DDP )) { +dev_put (netdev); +return -EINVAL ;EINVAL or ENODEV?
ENODEV seems more appropriate, we'll use it. Thanks
Quoted 21 lines
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ } + + ret = netdev->tcp_ddp_ops->tcp_ddp_sk_add (netdev, + queue->sock->sk, + ddp_config); +if (!ret) +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; +else +dev_put (netdev);drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 272
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+return ret; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__);Same here.
Quoted 7 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 278
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+return ; + } + + netdev->tcp_ddp_ops->tcp_ddp_sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ;Just a general question, why is this needed?
This assignment is symmetric with the nvme_tcp_offload_socket assignment. The idea was to ensure that the functions established during offload cannot be used from this moment.
Quoted 30 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 291
Quoted 13 lines
drivers/nvme/host/tcp.c +13
-0+dev_put (netdev);/* put the queue_init get_netdev_for_sock() */ +} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +struct net_device *netdev =get_netdev_for_sock (queue->sock->sk, true); +int ret =0 ; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ;Same here
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 304
Quoted 13 lines
drivers/nvme/host/tcp.c +13
-0+ } + +if (netdev->features &NETIF_F_HW_TCP_DDP && + netdev->tcp_ddp_ops && + netdev->tcp_ddp_ops->tcp_ddp_limits) + ret = netdev->tcp_ddp_ops->tcp_ddp_limits (netdev, limits); +else + ret = -EOPNOTSUPP ; + +if (!ret) { + queue->ctrl->offloading_netdev = netdev; +pr_info ("%s netdev %s offload limits: max_ddp_sgl_len %d\n" , + __func__, netdev->name, limits->max_ddp_sgl_len);
dev_info, and given that it per-queue, please make itdev_dbg.
Quoted 6 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 309
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); + }else { + queue->ctrl->offloading_netdev =NULL ;Maybe nullify in the controller setup intead?
It is already set to zero after allocation (i.e., kzalloc). The goal here is to ensure it is zero in case there is a reset and it was non zero due to offload being used in the past.
Quoted 12 lines
Quoted 12 lines
drivers/nvme/host/tcp.c +12
-0+ } + +dev_put (netdev); + +return ret; +} + +static +void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev;
Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 327
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+u64 resync_val; +u32 resync_seq; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return ;What happens now, fallback to SW? Maybe
dev_warnthen.. will the SW keep seeing these responses after one failed?
As long as there is no resync response the system falls back to software, and this message is emitted. I'll move it to below to display it only when it is relevant and not every time this function is called.
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 336
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ } + + resync_val =atomic64_read (&queue->resync_req); +if ((resync_val &TCP_DDP_RESYNC_REQ ) ==0 ) +return ; + + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ;I think it will be better to pass the skb to this func and keep the
pdu_seqcontained locally.
This requires passing the offset to obtain the sequence:
u64 pdu_seq = TCP_SKB_CB(skb)->seq + *offset - queue->pdu_offset;
It makes the interface a bit ugly, IMO.
Quoted 56 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 339
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ +if (atomic64_cmpxchg (&queue->resync_req, resync_val, (resync_val -1 ))) + netdev->tcp_ddp_ops->tcp_ddp_resync (netdev, queue->sock->sk, pdu_seq);A small comment on this manipulation may help the reader.
Quoted 51 lines
drivers/nvme/host/tcp.c +44
-0+} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +atomic64_set (&queue->resync_req, + (((uint64_t )seq <<32 ) | flags)); + +return true; +} + +#else + +static +int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue, +struct nvme_tcp_config *config) +{ +return -EINVAL ; +} + +static +void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{} + +static +int nvme_tcp_offload_limits (struct nvme_tcp_queue *queue, +struct tcp_ddp_limits *limits) +{ +return -EINVAL ; +} + +static +void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +unsigned int pdu_seq) +{} + +bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +return false; +} + +#endif +static void nvme_tcp_init_iter (struct nvme_tcp_request *req,unsigned int dir) {@@ -628,6 +786,11 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret;
Quoted 5 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 792
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+u64 pdu_seq =TCP_SKB_CB (skb)->seq + *offset - queue->pdu_offset; + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_resync_response (queue, pdu_seq);Here, just pass in (queue, skb)
Do you mean (queue, skb, offset)?
We need the offset for the pdu_seq calculation above.
Quoted 20 lines
Quoted 14 lines
drivers/nvme/host/tcp.c +3
-0+ ret =skb_copy_bits (skb, *offset, &pdu[queue->pdu_offset], rcv_len);if (unlikely (ret))@@ -1370,6 +1533,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[qid]; +struct nvme_tcp_config config; +struct tcp_ddp_limits limits;int ret, rcv_pdu_size; queue->ctrl = ctrl;drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1655
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0@@ -1487,6 +1652,26 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl,#endif write_unlock_bh (&queue->sock->sk->sk_callback_lock); +if (nvme_tcp_queue_id (queue) !=0 ) {if (!nvme_tcp_admin_queue(queue)) {
Quoted 11 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1665
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ config.cfg.type =TCP_DDP_NVME ; + config.pfv =NVME_TCP_PFV_1_0 ; + config.cpda =0 ; + config.dgst = queue->hdr_digest ? +NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.dgst |= queue->data_digest ? +NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.queue_size = queue->queue_size; + config.queue_id =nvme_tcp_queue_id (queue); + config.io_cpu = queue->io_cpu;Can the config initialization move to
nvme_tcp_offload_socket?
Definitely. The original idea of placing it here was that the nvme-tcp handshake may influence the parameters. But, at this moment, I realize that it is orthogonal.
Quoted 11 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1672
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ + ret =nvme_tcp_offload_socket (queue, &config); +if (!ret) +set_bit (NVME_TCP_Q_OFFLOADS , &queue->flags); + }else { + ret =nvme_tcp_offload_limits (queue, &limits); + }I'm thinking that instead of this conditional, we want to place nvme
nvme_tcp_alloc_admin_queueinnvme_tcp_alloc_admin_queue, and also movenvme_tcp_alloc_admin_queueto__nvme_tcp_alloc_io_queuesloop.
We need to do it only after the socket is connected to obtain the 5-tuple and the appropriate netdev, and preferably after the protocol handshake, as there is nothing to offload there.
Quoted 6 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, · line 1673
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+/* offload is opportunistic - failure is non-critical */ Than make it void...
Quoted 3 lines
drivers/nvme/host/tcp.c +1
-0+return 0 ;
Quoted 9 lines
drivers/nvme/host/tcp.c static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) · line 1709
Quoted 8 lines
drivers/nvme/host/tcp.c +3
-0err_init_connect:@@ -1519,6 +1704,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_calls (queue);cancel_work_sync (&queue->io_work); + +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_unoffload_socket (queue);Why not in
nvme_tcp_free_queue, symmetric to the alloc?
Sure. I've tried to keep it close to the socket disconnect, which isn't symmetrical for some reason, which suggests that these are interchangable?!
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +0
-0}static void nvme_tcp_stop_queue (struct nvme_ctrl *nctrl,int qid)include/linux/nvme-tcp.h +0
-0@@ -8,6 +8,8 @@#define _LINUX_NVME_TCP_H
Quoted 4 lines
include/linux/nvme-tcp.h line 12
Quoted 3 lines
include/linux/nvme-tcp.h +2
-0#include <linux/nvme.h> +#include <net/sock.h> +#include <net/tcp_ddp.h> Why is this needed? I think we want to place this in
tcp.cno?
Not needed. It's probably leftover from previous iterations on the code. Removed for the next iteration of the patchset.
A message that is not in the archive (sent off-list or not yet received): PH0PR18MB3845430DDF572E0DD4832D06CCED0@PH0PR18MB3845.namprd18.prod.outlook.com
RE: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path
Quoted 38 lines · Sagi Grimberg
Quoted 10 lines · Boris Pismenny
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the tcp_ddp_sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload 2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP toconfirm
Quoted 16 lines
it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then fiding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (tcp_ddp_resync), which will update the HW, and resume offload when all is successful.Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
tcp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Quoted 6 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 188+++++++++++++++++++++++++++++++++++++++
Quoted 3 lines
include/linux/nvme-tcp.h | 2 + 2 files changed, 190 insertions(+)
Quoted 9 lines
drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65
Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0
-0
8f4f29f18b8c..06711ac095f2100644drivers/nvme/host/tcp.c +1
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3,
Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a TCP_OFFLOAD transport type would be added, we are concerned that perhaps the generic term "offload" for both the transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g. NVME_TCP_Q_DDP or NVME_TCP_Q_DIRECT? Also, no need to quote the entire patch. Just a few lines above your response like I did here.
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path
Quoted 12 lines
Quoted 10 lines
drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65
Quoted 10 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0
-0
8f4f29f18b8c..06711ac095f2100644drivers/nvme/host/tcp.c +1
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3,Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a
TCP_OFFLOADtransport type would be added,
We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.
Quoted 1 line
we are concerned that perhaps the generic term "offload" for both the transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g.
NVME_TCP_Q_DDPorNVME_TCP_Q_DIRECT?
We can call this NVME_TCP_Q_DDP, no issues with that.
RFC PATCH 5/10 nvme-tcp: Add DDP offload control path
Quoted 14 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0
-0
8f4f29f18b8c..06711ac095f2100644drivers/nvme/host/tcp.c +0
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {Quoted 8 lines
Quoted 4 lines
drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3,Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a
TCP_OFFLOADtransport type would be added,
Quoted 4 lines
We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.
We plan to do this soon.
Quoted 8 lines
Quoted 1 line
we are concerned that perhaps the generic term "offload" for both the
transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g.
NVME_TCP_Q_DDPorNVME_TCP_Q_DIRECT?We can call this
NVME_TCP_Q_DDP, no issues with that.
Great. Thanks.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
RE: [PATCH net-next RFC v1 05/10] nvme-tcp: Add DDP offload control path
Quoted 15 lines · Sagi Grimberg
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/tcp.c +0
-0index
8f4f29f18b8c..06711ac095f2100644drivers/nvme/host/tcp.c +0
-0@@ -62,6 +62,7 @@ enum nvme_tcp_queue_flags {Quoted 8 lines
Quoted 4 lines
drivers/nvme/host/tcp.c enum nvme_tcp_queue_flags { · line 65
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFFLOADS = 3,Sagi - following our discussion and your suggestions regarding the NVMeTCP Offload ULP module that we are working on at Marvell in which a
TCP_OFFLOADtransport type would be added,
Quoted 4 lines
We still need to see how this pans out.. it's hard to predict if this is the best approach before seeing the code. I'd suggest to share some code so others can share their input.
We plan to do this soon.
Quoted 9 lines
Quoted 2 lines
we are concerned that perhaps the generic term "offload" for both the
transport type (for the Marvell work) and for the DDP and CRC offload queue (for the Mellanox work) may be misleading and confusing to developers and to users. Perhaps the naming should be "direct data placement", e.g.
NVME_TCP_Q_DDPorNVME_TCP_Q_DIRECT?We can call this
NVME_TCP_Q_DDP, no issues with that.
Great. Thanks.
RFC PATCH 6/10 nvme-tcp: Add DDP data-path
In reply to Boris PismennyIntroduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id in the PDU.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.
The data-path interface contains two routines: tcp_ddp_setup/teardown. The setup provides the mapping from command_id to the request buffers, while the teardown removes this mapping.
For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the generic mq_blk completion.
--- drivers/nvme/host/tcp.c | 121 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 117 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +117 -4
@@ -56,6 +56,11 @@ struct nvme_tcp_request {size_t offset;size_t data_sent;enum nvme_tcp_send_state state; + +bool offloaded; +struct tcp_ddp_io ddp; +__le16 status; +union nvme_result result; };enum nvme_tcp_queue_flags {
@@ -231,10 +236,76 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,#ifdef CONFIG_TCP_DDP bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +void nvme_tcp_ddp_teardown_done (void *ddp_ctx);const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops__read_mostly = { + .resync_request = nvme_tcp_resync_request, + .ddp_teardown_done = nvme_tcp_ddp_teardown_done, }; +static +int nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct net_device *netdev = queue->ctrl->offloading_netdev; +int ret; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ; + } + + ret = netdev->tcp_ddp_ops->tcp_ddp_teardown (netdev, queue->sock->sk, + &req->ddp, rq); +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); + req->offloaded = false; +return ret; +} + +void nvme_tcp_ddp_teardown_done (void *ddp_ctx) +{ +struct request *rq = ddp_ctx; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (!nvme_try_complete_req (rq,cpu_to_le16 (req->status <<1 ), req->result)) +nvme_complete_rq (rq); +} + +static +int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct net_device *netdev = queue->ctrl->offloading_netdev; +int ret; + + req->offloaded = false; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__); +return -EINVAL ; + } + + req->ddp.command_id = command_id; + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret =sg_alloc_table_chained (&req->ddp.sg_table, +blk_rq_nr_phys_segments (rq), req->ddp.sg_table.sgl, +SG_CHUNK_SIZE ); +if (ret) +return -ENOMEM ; + req->ddp.nents =blk_rq_map_sg (rq->q, rq, req->ddp.sg_table.sgl); + + ret = netdev->tcp_ddp_ops->tcp_ddp_setup (netdev, + queue->sock->sk, + &req->ddp); +if (!ret) + req->offloaded = true; +return ret; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue,struct nvme_tcp_config *config)
@@ -351,6 +422,25 @@ bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)#else +static +int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +return -EINVAL ; +} + +static +int nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +return -EINVAL ; +} + +void nvme_tcp_ddp_teardown_done (void *ddp_ctx) +{} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue,struct nvme_tcp_config *config)
@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)static int nvme_tcp_process_nvme_cqe (struct nvme_tcp_queue *queue,struct nvme_completion *cqe) { +struct nvme_tcp_request *req;struct request *rq; rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), cqe->command_id);
@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,return -EINVAL ; } -if (!nvme_try_complete_req (rq, cqe->status, cqe->result)) -nvme_complete_rq (rq); + req =blk_mq_rq_to_pdu (rq); +if (req->offloaded) { + req->status = cqe->status; + req->result = cqe->result; +nvme_tcp_teardown_ddp (queue, cqe->command_id, rq); + }else { +if (!nvme_try_complete_req (rq, cqe->status, cqe->result)) +nvme_complete_rq (rq); + } queue->nr_cqe++;return 0 ;
@@ -836,9 +934,18 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,static inline void nvme_tcp_end_request (struct request *rq,u16 status) {union nvme_result res = {}; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_tcp_queue *queue = req->queue; +struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu; -if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) -nvme_complete_rq (rq); +if (req->offloaded) { + req->status =cpu_to_le16 (status <<1 ); + req->result = res; +nvme_tcp_teardown_ddp (queue, pdu->command_id, rq); + }else { +if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) +nvme_complete_rq (rq); + } } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset; +struct request *rq =blk_mq_rq_from_pdu (req);int flags =MSG_DONTWAIT ;int ret;
@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else flags |=MSG_EOR ; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && +blk_rq_nr_phys_segments (rq) &&rq_data_dir (rq) ==READ ) +nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, rq); +if (queue->hdr_digest && !req->offset)nvme_tcp_hdgst (queue->snd_hash, pdu,sizeof (*pdu));
@@ -2448,6 +2560,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, req->data_len =blk_rq_nr_phys_segments (rq) ?blk_rq_payload_bytes (rq) :0 ; req->curr_bio = rq->bio; + req->offloaded = false;if (rq_data_dir (rq) ==WRITE && req->data_len <=nvme_tcp_inline_data_size (queue))
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path
Quoted 10 lines
bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);drivers/nvme/host/tcp.c +4
-0+void nvme_tcp_ddp_teardown_done (void *ddp_ctx);const struct tcp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops__read_mostly = { + .resync_request = nvme_tcp_resync_request, + .ddp_teardown_done = nvme_tcp_ddp_teardown_done, }; +static
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 256
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+int nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct net_device *netdev = queue->ctrl->offloading_netdev; +int ret; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__);
dev_info_ratelimited
Quoted 32 lines
drivers/nvme/host/tcp.c +32
-0+return -EINVAL ; + } + + ret = netdev->tcp_ddp_ops->tcp_ddp_teardown (netdev, queue->sock->sk, + &req->ddp, rq); +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); + req->offloaded = false; +return ret; +} + +void nvme_tcp_ddp_teardown_done (void *ddp_ctx) +{ +struct request *rq = ddp_ctx; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (!nvme_try_complete_req (rq,cpu_to_le16 (req->status <<1 ), req->result)) +nvme_complete_rq (rq); +} + +static +int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct net_device *netdev = queue->ctrl->offloading_netdev; +int ret; + + req->offloaded = false; + +if (unlikely (!netdev)) { +pr_info_ratelimited ("%s: netdev not found\n" , __func__);
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 298
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+return -EINVAL ; + } + + req->ddp.command_id = command_id; + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret =sg_alloc_table_chained (&req->ddp.sg_table, +blk_rq_nr_phys_segments (rq), req->ddp.sg_table.sgl, +SG_CHUNK_SIZE ); +if (ret) +return -ENOMEM ;
newline here
Quoted 95 lines
drivers/nvme/host/tcp.c +51
-4+ req->ddp.nents =blk_rq_map_sg (rq->q, rq, req->ddp.sg_table.sgl); + + ret = netdev->tcp_ddp_ops->tcp_ddp_setup (netdev, + queue->sock->sk, + &req->ddp); +if (!ret) + req->offloaded = true; +return ret; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue,struct nvme_tcp_config *config)@@ -351,6 +422,25 @@ bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)#else +static +int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +return -EINVAL ; +} + +static +int nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +uint16_t command_id, +struct request *rq) +{ +return -EINVAL ; +} + +void nvme_tcp_ddp_teardown_done (void *ddp_ctx) +{} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue,struct nvme_tcp_config *config)@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)static int nvme_tcp_process_nvme_cqe (struct nvme_tcp_queue *queue,struct nvme_completion *cqe) { +struct nvme_tcp_request *req;struct request *rq; rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), cqe->command_id);@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,return -EINVAL ; } -if (!nvme_try_complete_req (rq, cqe->status, cqe->result)) -nvme_complete_rq (rq); + req =blk_mq_rq_to_pdu (rq); +if (req->offloaded) { + req->status = cqe->status; + req->result = cqe->result; +nvme_tcp_teardown_ddp (queue, cqe->command_id, rq); + }else { +if (!nvme_try_complete_req (rq, cqe->status, cqe->result)) +nvme_complete_rq (rq); + } queue->nr_cqe++;return 0 ;@@ -836,9 +934,18 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,static inline void nvme_tcp_end_request (struct request *rq,u16 status) {union nvme_result res = {}; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_tcp_queue *queue = req->queue; +struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu; -if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) -nvme_complete_rq (rq); +if (req->offloaded) { + req->status =cpu_to_le16 (status <<1 ); + req->result = res; +nvme_tcp_teardown_ddp (queue, pdu->command_id, rq); + }else { +if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) +nvme_complete_rq (rq); + } } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset; +struct request *rq =blk_mq_rq_from_pdu (req);int flags =MSG_DONTWAIT ;int ret;
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else flags |=MSG_EOR ; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && +blk_rq_nr_phys_segments (rq) &&rq_data_dir (rq) ==READ ) +nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, rq);
I'd assume that this is something we want to setup in nvme_tcp_setup_cmd_pdu. Why do it here?
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path
Quoted 19 lines
Quoted 19 lines
static
intnvme_tcp_offload_socket(struct nvme_tcp_queue *queue, struct nvme_tcp_config *config)drivers/nvme/host/tcp.c +2
-2@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, struct nvme_completion *cqe) { + struct nvme_tcp_request *req; struct request *rq; rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(structnvme_tcp_queue *queue, return -EINVAL; } - if (!nvme_try_complete_req(rq, cqe->status, cqe->result)) - nvme_complete_rq(rq); + req = blk_mq_rq_to_pdu(rq);
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · line 743
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ if (req->offloaded) { + req->status = cqe->status; + req->result = cqe->result; + nvme_tcp_teardown_ddp(queue, cqe->command_id, rq); + } else { + if (!nvme_try_complete_req(rq, cqe->status, cqe->result)) + nvme_complete_rq(rq); + }
Oh forgot to ask,
We have places in the driver that we may complete (cancel) one or more requests from the error recovery or timeout flow. We first prevent future incoming RX on the socket such that we can safely cancel requests. This may break with the deferred completion in ddp_teardown_done.
If I have a request that is waiting for ddp_teardown_done do I have a way to tell the HW to never call ddp_teardown_done on a specific socket?
If so the place to is in nvme_tcp_stop_queue.
Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path
Quoted 34 lines · Sagi Grimberg
Quoted 18 lines
Quoted 18 lines
static
intnvme_tcp_offload_socket(struct nvme_tcp_queue *queue, struct nvme_tcp_config *config)drivers/nvme/host/tcp.c +2
-2@@ -630,6 +720,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, struct nvme_completion *cqe) { + struct nvme_tcp_request *req; struct request *rq; rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);@@ -641,8 +732,15 @@ static int nvme_tcp_process_nvme_cqe(structnvme_tcp_queue *queue, return -EINVAL; } - if (!nvme_try_complete_req(rq, cqe->status, cqe->result)) - nvme_complete_rq(rq); + req = blk_mq_rq_to_pdu(rq);Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · line 743
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ if (req->offloaded) { + req->status = cqe->status; + req->result = cqe->result; + nvme_tcp_teardown_ddp(queue, cqe->command_id, rq); + } else { + if (!nvme_try_complete_req(rq, cqe->status, cqe->result)) + nvme_complete_rq(rq); + }Oh forgot to ask,
We have places in the driver that we may complete (cancel) one or more requests from the error recovery or timeout flow. We first prevent future incoming RX on the socket such that we can safely cancel requests. This may break with the deferred completion in
ddp_teardown_done.
Quoted 5 lines
If I have a request that is waiting for
ddp_teardown_donedo I have a way to tell the HW to never callddp_teardown_doneon a specific socket?If so the place to is in
nvme_tcp_stop_queue.
Interesting and indeed, it is a problem that we haven't considered.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path
In reply to Sagi GrimbergQuoted 14 lines · Sagi Grimberg
Quoted 14 lines
static int nvme_tcp_recv_data(struct
nvme_tcp_queue*queue, structsk_buff*skb,drivers/nvme/host/tcp.c +1
-0@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset; +struct request *rq =blk_mq_rq_from_pdu (req);int flags =MSG_DONTWAIT ;int ret;@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else flags |=MSG_EOR ;
Quoted 5 lines
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && +blk_rq_nr_phys_segments (rq) &&rq_data_dir (rq) ==READ ) +nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, rq);I'd assume that this is something we want to setup in
nvme_tcp_setup_cmd_pdu. Why do it here?
Our goal in placing it here is to keep both setup and teardown in the same thread. This enables drivers to avoid locking for per-queue operations.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 06/10] nvme-tcp: Add DDP data-path
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
static int nvme_tcp_recv_data(struct
nvme_tcp_queue*queue, structsk_buff*skb,drivers/nvme/host/tcp.c +1
-0@@ -1115,6 +1222,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)bool inline_data =nvme_tcp_has_inline_data (req);u8 hdgst =nvme_tcp_hdgst_len (queue);int len =sizeof (*pdu) + hdgst - req->offset; +struct request *rq =blk_mq_rq_from_pdu (req);int flags =MSG_DONTWAIT ;int ret;@@ -1123,6 +1231,10 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else flags |=MSG_EOR ;
Quoted 7 lines
Quoted 5 lines
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1236
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && +blk_rq_nr_phys_segments (rq) &&rq_data_dir (rq) ==READ ) +nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, rq);I'd assume that this is something we want to setup in
nvme_tcp_setup_cmd_pdu. Why do it here?Our goal in placing it here is to keep both setup and teardown in the same thread. This enables drivers to avoid locking for per-queue operations.
I also think that it is cleaner when setting up the PDU. Do note that if queues match 1x1 with cpu cores then any synchronization is pretty lightweight, and if not, we have other synchronizations anyways...
RFC PATCH 7/10 nvme-tcp : Recalculate crc in the end of the capsule
In reply to Boris PismennyFrom: Yoray Zack <[email protected]>
crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.
This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.
--- drivers/nvme/host/tcp.c | 66 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +58 -8
@@ -94,6 +94,7 @@ struct nvme_tcp_queue {size_t data_remaining;size_t ddgst_remaining;unsigned int nr_cqe; +bool crc_valid;/* send state */ struct nvme_tcp_request *request;
@@ -233,6 +234,41 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +static inline bool nvme_tcp_device_ddgst_ok (struct nvme_tcp_queue *queue) +{ +return queue->crc_valid; +} + +static inline void nvme_tcp_device_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb) +{ + if (queue->crc_valid) +#ifdef CONFIG_TCP_DDP_CRC + queue->crc_valid = skb->ddp_crc; +#else + queue->crc_valid = false; +#endif +} + +static void nvme_tcp_crc_recalculate (struct nvme_tcp_queue *queue, +struct nvme_tcp_data_pdu *pdu) +{ +struct nvme_tcp_request *req; +struct request *rq; + + rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), pdu->command_id); +if (!rq) +return ; + req =blk_mq_rq_to_pdu (rq); +crypto_ahash_init (queue->rcv_hash); + req->ddp.sg_table.sgl = req->ddp.first_sgl; +/* req->ddp.sg_table is allocated and filled in nvme_tcp_setup_ddp */ +ahash_request_set_crypt (queue->rcv_hash, req->ddp.sg_table.sgl,NULL , +le32_to_cpu (pdu->data_length)); +crypto_ahash_update (queue->rcv_hash); +} + +#ifdef CONFIG_TCP_DDP bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags);
@@ -706,6 +742,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->pdu_offset =0 ; queue->data_remaining =-1 ; queue->ddgst_remaining =0 ; + queue->crc_valid = true; }static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl)
@@ -955,6 +992,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,struct nvme_tcp_request *req;struct request *rq; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_device_ddgst_update (queue, skb); rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), pdu->command_id);if (!rq) { dev_err(queue->ctrl->ctrl.device,
@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len =min_t (size_t, recv_len,iov_iter_count (&req->iter)); -if (queue->data_digest) +if (queue->data_digest && !test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) ret =skb_copy_and_hash_datagram_iter (skb, *offset, &req->iter, recv_len, queue->rcv_hash);else
@@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,if (!queue->data_remaining) {if (queue->data_digest) { -nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst); queue->ddgst_remaining =NVME_TCP_DIGEST_LENGTH ; }else {if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {
@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; +bool ddgst_offload_fail;int ret; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_device_ddgst_update (queue, skb); ret =skb_copy_bits (skb, *offset, &ddgst[off], recv_len);if (unlikely (ret))return ret;
@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,if (queue->ddgst_remaining)return 0 ; -if (queue->recv_ddgst != queue->exp_ddgst) { -dev_err (queue->ctrl->ctrl.device, -"data digest error: recv %#x expected %#x\n" , -le32_to_cpu (queue->recv_ddgst), -le32_to_cpu (queue->exp_ddgst)); -return -EIO ; + ddgst_offload_fail = !nvme_tcp_device_ddgst_ok (queue); +if (!test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) || + ddgst_offload_fail) { +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && + ddgst_offload_fail) +nvme_tcp_crc_recalculate (queue, pdu); + +nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst); +if (queue->recv_ddgst != queue->exp_ddgst) { +dev_err (queue->ctrl->ctrl.device, +"data digest error: recv %#x expected %#x\n" , +le32_to_cpu (queue->recv_ddgst), +le32_to_cpu (queue->exp_ddgst)); +return -EIO ; + } }if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule
Quoted 3 lines
crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.
Can you clarify in the patch description that this is only for pdu data digest and not header digest?
Quoted 5 lines
This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.
??
From my scan it doeesn't look like you do that.. Am I missing something? Can you explain?
Quoted 22 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 66 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 8 deletions(-)drivers/nvme/host/tcp.c +1
-0@@ -94,6 +94,7 @@ struct nvme_tcp_queue {size_t data_remaining;size_t ddgst_remaining;unsigned int nr_cqe; +bool crc_valid;/* send state */ struct nvme_tcp_request *request;
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 237
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0@@ -233,6 +234,41 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +static inline bool nvme_tcp_device_ddgst_ok (struct nvme_tcp_queue *queue)
Maybe call it nvme_tcp_ddp_ddgst_ok?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 243
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+{ +return queue->crc_valid; +} + +static inline void nvme_tcp_device_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb)
Maybe call it nvme_tcp_ddp_ddgst_update?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 254
Quoted 11 lines
drivers/nvme/host/tcp.c +11
-0+{ + if (queue->crc_valid) +#ifdef CONFIG_TCP_DDP_CRC + queue->crc_valid = skb->ddp_crc; +#else + queue->crc_valid = false; +#endif +} + +static void nvme_tcp_crc_recalculate (struct nvme_tcp_queue *queue, +struct nvme_tcp_data_pdu *pdu)
Maybe call it nvme_tcp_ddp_ddgst_recalc?
Quoted 29 lines
drivers/nvme/host/tcp.c +18
-0+{ +struct nvme_tcp_request *req; +struct request *rq; + + rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), pdu->command_id); +if (!rq) +return ; + req =blk_mq_rq_to_pdu (rq); +crypto_ahash_init (queue->rcv_hash); + req->ddp.sg_table.sgl = req->ddp.first_sgl; +/* req->ddp.sg_table is allocated and filled in nvme_tcp_setup_ddp */ +ahash_request_set_crypt (queue->rcv_hash, req->ddp.sg_table.sgl,NULL , +le32_to_cpu (pdu->data_length)); +crypto_ahash_update (queue->rcv_hash); +} + +#ifdef CONFIG_TCP_DDP bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags);@@ -706,6 +742,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->pdu_offset =0 ; queue->data_remaining =-1 ; queue->ddgst_remaining =0 ; + queue->crc_valid = true; }static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl)
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 996
Quoted 5 lines
drivers/nvme/host/tcp.c +2
-0@@ -955,6 +992,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,struct nvme_tcp_request *req;struct request *rq; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_device_ddgst_update (queue, skb);
Is the queue->data_digest condition missing here?
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1036
Quoted 11 lines
drivers/nvme/host/tcp.c +1
-1rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), pdu->command_id);if (!rq) { dev_err(queue->ctrl->ctrl.device,@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len =min_t (size_t, recv_len,iov_iter_count (&req->iter)); -if (queue->data_digest) +if (queue->data_digest && !test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) ret =skb_copy_and_hash_datagram_iter (skb, *offset, &req->iter, recv_len, queue->rcv_hash);
This is the skb copy and hash, not clear why you say that you move this to the end...
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1015
Quoted 6 lines
drivers/nvme/host/tcp.c +0
-1else @@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,if (!queue->data_remaining) {if (queue->data_digest) { -nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst);
If I instead do:
if (!test_bit(NVME_TCP_Q_OFFLOADS,
&queue->flags))
nvme_tcp_ddgst_final(queue->rcv_hash,
&queue->exp_ddgst);Does that help the mess in nvme_tcp_recv_ddgst?
Quoted 29 lines
drivers/nvme/host/tcp.c +7
-6queue->ddgst_remaining =NVME_TCP_DIGEST_LENGTH ; }else {if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; +bool ddgst_offload_fail;int ret; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_device_ddgst_update (queue, skb); ret =skb_copy_bits (skb, *offset, &ddgst[off], recv_len);if (unlikely (ret))return ret;@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,if (queue->ddgst_remaining)return 0 ; -if (queue->recv_ddgst != queue->exp_ddgst) { -dev_err (queue->ctrl->ctrl.device, -"data digest error: recv %#x expected %#x\n" , -le32_to_cpu (queue->recv_ddgst), -le32_to_cpu (queue->exp_ddgst)); -return -EIO ; + ddgst_offload_fail = !nvme_tcp_device_ddgst_ok (queue); +if (!test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) || + ddgst_offload_fail) { + if (test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags) &&
drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1102
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ ddgst_offload_fail) +nvme_tcp_crc_recalculate (queue ,pdu ); + +nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst); +if (queue->recv_ddgst != queue->exp_ddgst) { +dev_err (queue->ctrl->ctrl.device, +"data digest error: recv %#x expected %#x\n" , +le32_to_cpu (queue->recv_ddgst), +le32_to_cpu (queue->exp_ddgst)); +return -EIO ;
This gets convoluted here...
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-0+ } }if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {
A message that is not in the archive (sent off-list or not yet received): PH0PR18MB3845764B48FD24C87FA34304CCED0@PH0PR18MB3845.namprd18.prod.outlook.com
A message that is not in the archive (sent off-list or not yet received): PH0PR18MB38458FD325BD77983D2623D4CCEB0@PH0PR18MB3845.namprd18.prod.outlook.com
RE: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule
Quoted 8 lines · Sagi Grimberg
Quoted 2 lines · Boris Pismenny
crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.
Can you clarify in the patch description that this is only for pdu data digest and not header digest?
Not a security expert, but according to my understanding, the NVMeTCP data digest is a layer 5 CRC, and as such it is expected to be end-to-end, meaning it is computed by layer 5 on the transmitter and verified on layer 5 on the receiver.
Any data corruption which happens in any of the lower layers, including their software processing, should be protected by this CRC. For example, if the IP or TCP stack has a bug that corrupts the NVMeTCP payload data, the CRC should protect against it. It seems that may not be the case with this offload.
Quoted 21 lines
Quoted 5 lines
This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.
??
From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?Quoted 7 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 66++++++++++++++++++++++++++++++++++++-----
Quoted 2 lines
1 file changed, 58 insertions(+), 8 deletions(-)
Quoted 9 lines
drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 97
Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0
-0
7bd97f856677..9a620d1dacb4100644drivers/nvme/host/tcp.c +1
-0@@ -94,6 +94,7 @@ struct nvme_tcp_queue {size_t data_remaining;size_t ddgst_remaining;unsigned int nr_cqe; +bool crc_valid;
I suggest to rename it to ddgst_valid.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule
Quoted 10 lines · Shai Malin
Quoted 7 lines · Sagi Grimberg
Quoted 2 lines · Boris Pismenny
crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.
Can you clarify in the patch description that this is only for pdu data digest and not header digest?
Not a security expert, but according to my understanding, the NVMeTCP data digest is a layer 5 CRC, and as such it is expected to be end-to-end, meaning it is computed by layer 5 on the transmitter and verified on layer 5 on the receiver.
Any data corruption which happens in any of the lower layers, including their software processing, should be protected by this CRC. For example, if the IP or TCP stack has a bug that corrupts the NVMeTCP payload data, the CRC should protect against it. It seems that may not be the case with this offload.
If the TCP/IP stack corrupts packet data, then likely many TCP/IP consumers will be effected, and it will be fixed promptly. Unlike with TOE, these bugs can be easily fixed/hotpatched by the community.
Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.
??
From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?Quoted 6 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 66++++++++++++++++++++++++++++++++++++-----
Quoted 2 lines
1 file changed, 58 insertions(+), 8 deletions(-)
Quoted 11 lines
Quoted 9 lines
drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 97
Quoted 9 lines
drivers/nvme/host/tcp.c index renamed from drivers/nvme/host/tcp.c +0
-0
7bd97f856677..9a620d1dacb4100644drivers/nvme/host/tcp.c +1
-0@@ -94,6 +94,7 @@ struct nvme_tcp_queue {size_t data_remaining;size_t ddgst_remaining;unsigned int nr_cqe; +bool crc_valid;I suggest to rename it to
ddgst_valid.
Sure
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH net-next RFC v1 07/10] nvme-tcp : Recalculate crc in the end of the capsule
In reply to Sagi GrimbergQuoted 4 lines · Sagi Grimberg
Quoted 2 lines
crc offload of the nvme capsule. Check if all the skb bits are on, and if not recalculate the crc in SW and check it.
Can you clarify in the patch description that this is only for pdu data digest and not header digest?
Will do
Quoted 9 lines
Quoted 5 lines
This patch reworks the receive-side crc calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload crc calculation.
??
From my scan it doeesn't look like you do that.. Am I missing something?
Can you explain?
The performance of CRC data digest in the offload's fallback path may be less good compared to CRC calculation with skb_copy_and_hash.
To be clear, the fallback path is occurs when queue->data_digest && test_bit(NVME_TCP_Q_OFF_CRC_RX, &queue->flags), while we receive SKBs where skb->ddp_crc = 0
Quoted 8 lines
Quoted 8 lines
rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id); if (!rq) { dev_err(queue->ctrl->ctrl.device,drivers/nvme/host/tcp.c +0
-0@@ -992,7 +1031,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len =min_t (size_t, recv_len,iov_iter_count (&req->iter));
Quoted 6 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1036
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-1-if (queue->data_digest) +if (queue->data_digest && !test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) ret =skb_copy_and_hash_datagram_iter (skb, *offset, &req->iter, recv_len, queue->rcv_hash);This is the skb copy and hash, not clear why you say that you move this to the end...
See the offload fallback path below
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1015
Quoted 7 lines
drivers/nvme/host/tcp.c +0
-1else @@ -1012,7 +1051,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,if (!queue->data_remaining) {if (queue->data_digest) { -nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst);If I instead do:
Quoted 6 lines
if (!test_bit(NVME_TCP_Q_OFFLOADS, &queue->flags)) nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);Does that help the mess in
nvme_tcp_recv_ddgst?
Not really, as the code path there takes care of the fallback path, i.e. offloaded requested, but didn't succeed.
Quoted 33 lines
Quoted 33 lines
drivers/nvme/host/tcp.c +10
-6queue->ddgst_remaining =NVME_TCP_DIGEST_LENGTH ; }else {if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {@@ -1033,8 +1071,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; +bool ddgst_offload_fail;int ret; +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags)) +nvme_tcp_device_ddgst_update (queue, skb); ret =skb_copy_bits (skb, *offset, &ddgst[off], recv_len);if (unlikely (ret))return ret;@@ -1045,12 +1086,21 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,if (queue->ddgst_remaining)return 0 ; -if (queue->recv_ddgst != queue->exp_ddgst) { -dev_err (queue->ctrl->ctrl.device, -"data digest error: recv %#x expected %#x\n" , -le32_to_cpu (queue->recv_ddgst), -le32_to_cpu (queue->exp_ddgst)); -return -EIO ; + ddgst_offload_fail = !nvme_tcp_device_ddgst_ok (queue); +if (!test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) || + ddgst_offload_fail) { +if (test_bit (NVME_TCP_Q_OFFLOADS , &queue->flags) && + ddgst_offload_fail) +nvme_tcp_crc_recalculate (queue, pdu); +
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1102
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst); +if (queue->recv_ddgst != queue->exp_ddgst) { +dev_err (queue->ctrl->ctrl.device, +"data digest error: recv %#x expected %#x\n" , +le32_to_cpu (queue->recv_ddgst), +le32_to_cpu (queue->exp_ddgst)); +return -EIO ;This gets convoluted here...
Will try to simplify, the general idea is that there are 3 paths with common code:
1. non-offload
2. offload failed
3. offload success
(1) and (2) share the code for finalizing checking the data digest, while (3) skips this entirely.
In other words, how about this:
```
offload_fail = !nvme_tcp_ddp_ddgst_ok(queue);
offload = test_bit(NVME_TCP_Q_OFF_CRC_RX, &queue->flags);
if (!offload || offload_fail) {
if (offload && offload_fail) // software-fallback
nvme_tcp_ddp_ddgst_recalc(queue, pdu);
nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst);
if (queue->recv_ddgst != queue->exp_ddgst) {
dev_err(queue->ctrl->ctrl.device,
"data digest error: recv %#x expected %#x\n",
le32_to_cpu(queue->recv_ddgst),
le32_to_cpu(queue->exp_ddgst));
return -EIO;
}
}
```Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/tcp.c +1
-0+ } }if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) {
RFC PATCH 8/10 nvme-tcp: Deal with netdevice DOWN events
In reply to Boris PismennyFrom: Or Gerlitz <[email protected]>
For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.
These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroyes them.
Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.
We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.
Since the recovery flow runs in a separate/dedicated WQ we need to wait in the notifier code for an ACK that all offloaded queues were stopped which means that the teardown queue offload ndo was called and the NIC doesn't have any resources related to that connection any more.
This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.
--- drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +37 -2
@@ -144,6 +144,7 @@ struct nvme_tcp_ctrl {static LIST_HEAD (nvme_tcp_ctrl_list);static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); +static struct notifier_block nvme_tcp_netdevice_nb;static struct workqueue_struct *nvme_tcp_wq;static const struct blk_mq_ops nvme_tcp_mq_ops;static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; queue->ctrl->ctrl.max_hw_sectors = limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); - }else { - queue->ctrl->offloading_netdev =NULL ; }dev_put (netdev);
@@ -1992,6 +1991,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) {int ret; +to_tcp_ctrl (ctrl)->offloading_netdev =NULL ; + ret =nvme_tcp_alloc_queue (ctrl,0 ,NVME_AQ_DEPTH );if (ret)return ret;
@@ -2885,6 +2886,26 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,return ERR_PTR (ret); } +static int nvme_tcp_netdev_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +struct net_device *ndev =netdev_notifier_info_to_dev (ptr); +struct nvme_tcp_ctrl *ctrl; + +switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev != ctrl->offloading_netdev) +continue ; +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq); + } +return NOTIFY_DONE ; +} +static struct nvmf_transport_ops nvme_tcp_transport = { .name ="tcp" , .module =THIS_MODULE ,
@@ -2899,13 +2920,26 @@ static struct nvmf_transport_ops nvme_tcp_transport = {static int __init nvme_tcp_init_module (void ) { +int ret; + nvme_tcp_wq =alloc_workqueue ("nvme_tcp_wq" ,WQ_MEM_RECLAIM |WQ_HIGHPRI ,0 );if (!nvme_tcp_wq)return -ENOMEM ; + nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event; + ret =register_netdevice_notifier (&nvme_tcp_netdevice_nb); +if (ret) { +pr_err ("failed to register netdev notifier\n" ); + goto out_err_reg_notifier; + } +nvmf_register_transport (&nvme_tcp_transport);return 0 ; + +out_err_reg_notifier: +destroy_workqueue (nvme_tcp_wq); +return ret; } static void __exit nvme_tcp_cleanup_module(void)
@@ -2913,6 +2947,7 @@ static void __exit nvme_tcp_cleanup_module(void)struct nvme_tcp_ctrl *ctrl;nvmf_unregister_transport (&nvme_tcp_transport); +unregister_netdevice_notifier (&nvme_tcp_netdevice_nb);mutex_lock (&nvme_tcp_ctrl_mutex);list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list)
Signature · 1 line
2.24.1
Sagi Grimberg
maintainerRe: [PATCH net-next RFC v1 08/10] nvme-tcp: Deal with netdevice DOWN events
Quoted 47 lines · Boris Pismenny
From: Or Gerlitz <[email protected]>
For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.
These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroyes them.
Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.
We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.
Since the recovery flow runs in a separate/dedicated WQ we need to wait in the notifier code for an ACK that all offloaded queues were stopped which means that the teardown queue offload ndo was called and the NIC doesn't have any resources related to that connection any more.
This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.
Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Yoray Zack <[email protected]>--- drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-)drivers/nvme/host/tcp.c +1
-0@@ -144,6 +144,7 @@ struct nvme_tcp_ctrl {static LIST_HEAD (nvme_tcp_ctrl_list);static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); +static struct notifier_block nvme_tcp_netdevice_nb;static struct workqueue_struct *nvme_tcp_wq;static const struct blk_mq_ops nvme_tcp_mq_ops;static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
drivers/nvme/host/tcp.c int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, · line 416
Quoted 5 lines
drivers/nvme/host/tcp.c +0
-2@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; queue->ctrl->ctrl.max_hw_sectors = limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); - }else { - queue->ctrl->offloading_netdev =NULL ;
Squash this change to the patch that introduced it.
Quoted 22 lines
drivers/nvme/host/tcp.c +8
-0}dev_put (netdev);@@ -1992,6 +1991,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) {int ret; +to_tcp_ctrl (ctrl)->offloading_netdev =NULL ; + ret =nvme_tcp_alloc_queue (ctrl,0 ,NVME_AQ_DEPTH );if (ret)return ret;@@ -2885,6 +2886,26 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,return ERR_PTR (ret); } +static int nvme_tcp_netdev_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +struct net_device *ndev =netdev_notifier_info_to_dev (ptr); +struct nvme_tcp_ctrl *ctrl; +
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 2904
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev != ctrl->offloading_netdev) +continue ; +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);
Worth a small comment that this we want the err_work to complete here. So if someone changes workqueues he may see this.
Quoted 7 lines
drivers/nvme/host/tcp.c +4
-0+ } +return NOTIFY_DONE ; +} +static struct nvmf_transport_ops nvme_tcp_transport = { .name ="tcp" , .module =THIS_MODULE ,
Re: [PATCH net-next RFC v1 08/10] nvme-tcp: Deal with netdevice DOWN events
Quoted 8 lines · Sagi Grimberg
drivers/nvme/host/tcp.c int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, · line 416
Quoted 6 lines
drivers/nvme/host/tcp.c +0
-2@@ -412,8 +413,6 @@ int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue, queue->ctrl->ctrl.max_segments = limits->max_ddp_sgl_len; queue->ctrl->ctrl.max_hw_sectors = limits->max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); - }else { - queue->ctrl->offloading_netdev =NULL ;Squash this change to the patch that introduced it.
OK, will look on that and I guess it should be fine to make this as you suggested
Quoted 10 lines
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 2904
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev != ctrl->offloading_netdev) +continue ; +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);
Quoted 2 lines
Worth a small comment that this we want the
err_workto complete here. So if someone changes workqueues he may see this.
ack
RFC PATCH 9/10 net/mlx5e: Add NVMEoTCP offload
In reply to Boris PismennyAdd NVMEoTCP offload to mlx5.
This patch implements the NVME-TCP offload added by previous commits. Similarly to other layer-5 offload, the offload is mapped to a TIR, and updated using WQEs.
- Use 128B CQEs when NVME-TCP offload is enabled
- Implement asynchronous ddp invalidaation by completing the nvme-tcp request only when the invalidate UMR is done
- Use KLM UMRs to implement ddp
- Use a dedicated icosq for all NVME-TCP work. This SQ is unique in the sense that it is driven directly by the NVME-TCP layer to submit and invalidate ddp requests.
- Add statistics for offload packets/bytes, ddp setup/teardown, and queue init/teardown
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 11 + .../net/ethernet/mellanox/mlx5/core/Makefile | 2 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 11 +- .../ethernet/mellanox/mlx5/core/en/params.h | 4 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 13 + .../mellanox/mlx5/core/en_accel/en_accel.h | 9 +- .../mellanox/mlx5/core/en_accel/fs_tcp.c | 10 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 894 ++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 116 +++ .../mlx5/core/en_accel/nvmeotcp_utils.h | 79 ++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 73 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 25 +- .../ethernet/mellanox/mlx5/core/en_stats.c | 26 + .../ethernet/mellanox/mlx5/core/en_stats.h | 16 + .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 6 + drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + include/linux/mlx5/device.h | 8 + include/linux/mlx5/mlx5_ifc.h | 121 ++- include/linux/mlx5/qp.h | 1 + 19 files changed, 1419 insertions(+), 12 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +11 -0
@@ -201,3 +201,14 @@ config MLX5_SW_STEERING default y help Build support for software-managed steering in the NIC. + +config MLX5_EN_NVMEOTCP + bool "NVMEoTCP accelaration" + depends on MLX5_CORE_EN + depends on TCP_DDP + depends on TCP_DDP_CRC + default y + help + Build support for NVMEoTCP accelaration in the NIC. + Note: Support for hardware with this capability needs to be selected + for this option to become available.
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -0
@@ -84,3 +84,5 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o steering/dr_ste.o steering/dr_send.o \ steering/dr_cmd.o steering/dr_fw.o \ steering/dr_action.o steering/fs_dr.o + +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +10 -1
@@ -209,7 +209,10 @@ struct mlx5e_umr_wqe {struct mlx5_wqe_ctrl_seg ctrl;struct mlx5_wqe_umr_ctrl_seg uctrl;struct mlx5_mkey_seg mkc; -struct mlx5_mtt inline_mtts[0 ]; +union { +struct mlx5_mtt inline_mtts[0 ]; +struct mlx5_klm inline_klms[0 ]; + }; };extern const char mlx5e_self_tests[][ETH_GSTRING_LEN ];
@@ -642,6 +645,9 @@ struct mlx5e_channel {struct mlx5e_xdpsq rq_xdpsq;struct mlx5e_txqsq sq[MLX5E_MAX_NUM_TC ];struct mlx5e_icosq icosq;/* internal control operations */ +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct mlx5e_icosq nvmeotcpsq;/* nvmeotcp umrs */ +#endif bool xdp;struct napi_struct napi;struct device *pdev;
@@ -815,6 +821,9 @@ struct mlx5e_priv {#endif #ifdef CONFIG_MLX5_EN_TLS struct mlx5e_tls *tls; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct mlx5e_nvmeotcp *nvmeotcp;#endif struct devlink_health_reporter *tx_reporter;struct devlink_health_reporter *rx_reporter;
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +4 -0
@@ -16,6 +16,7 @@ struct mlx5e_cq_param {struct mlx5_wq_param wq;u16 eq_ix;u8 cq_period_mode; +bool force_cqe128; };struct mlx5e_rq_param {
@@ -38,6 +39,9 @@ struct mlx5e_channel_param {struct mlx5e_sq_param xdp_sq;struct mlx5e_sq_param icosq;struct mlx5e_sq_param async_icosq; +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct mlx5e_sq_param nvmeotcpsq; +#endif }; static inline bool mlx5e_qid_get_ch_if_in_group(struct mlx5e_params *params,
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +13 -0
@@ -32,6 +32,11 @@ enum mlx5e_icosq_wqe_type { MLX5E_ICOSQ_WQE_SET_PSV_TLS, MLX5E_ICOSQ_WQE_GET_PSV_TLS,#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +MLX5E_ICOSQ_WQE_UMR_NVME_TCP , +MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE , + MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP, +#endif };/* General */
@@ -173,6 +178,14 @@ struct mlx5e_icosq_wqe_info {struct {struct mlx5e_ktls_rx_resync_buf *buf; } tls_get_params; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct { +struct mlx5e_nvmeotcp_queue *queue; + } nvmeotcp_q; +struct { +struct nvmeotcp_queue_entry *entry; + } nvmeotcp_qe;#endif }; };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +8 -1
@@ -39,6 +39,7 @@#include "en_accel/ipsec_rxtx.h" #include "en_accel/tls.h" #include "en_accel/tls_rxtx.h" +#include "en_accel/nvmeotcp.h" #include "en.h" #include "en/txrx.h"
@@ -162,11 +163,17 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,static inline int mlx5e_accel_init_rx (struct mlx5e_priv *priv) { -return mlx5e_ktls_init_rx (priv); +int tls, nvmeotcp; + + tls =mlx5e_ktls_init_rx (priv); + nvmeotcp =mlx5e_nvmeotcp_init_rx (priv); + +return tls && nvmeotcp; }static inline void mlx5e_accel_cleanup_rx (struct mlx5e_priv *priv) { +mlx5e_nvmeotcp_cleanup_rx (priv);mlx5e_ktls_cleanup_rx (priv); }#endif /* __MLX5E_EN_ACCEL_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +10 -0
@@ -14,6 +14,7 @@ enum accel_fs_tcp_type {struct mlx5e_accel_fs_tcp {struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES ];struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES ]; +refcount_t ref_count; };static enum mlx5e_traffic_types fs_accel2tt (enum accel_fs_tcp_type i)
@@ -335,6 +336,7 @@ static int accel_fs_tcp_enable(struct mlx5e_priv *priv)return err; } } +refcount_set (&priv->fs.accel_tcp->ref_count,1 );return 0 ; }
@@ -358,6 +360,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)if (!priv->fs.accel_tcp)return ; +if (!refcount_dec_and_test (&priv->fs.accel_tcp->ref_count)) +return ; +accel_fs_tcp_disable (priv);for (i =0 ; i <ACCEL_FS_TCP_NUM_TYPES ; i++)
@@ -374,6 +379,11 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)if (!MLX5_CAP_FLOWTABLE_NIC_RX (priv->mdev, ft_field_support.outer_ip_version))return -EOPNOTSUPP ; +if (priv->fs.accel_tcp) { +refcount_inc (&priv->fs.accel_tcp->ref_count); +return 0 ; + } + priv->fs.accel_tcp =kzalloc (sizeof (*priv->fs.accel_tcp),GFP_KERNEL );if (!priv->fs.accel_tcp)return -ENOMEM ;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c new file +894 -0
894 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h new file +116 -0
116 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h new file +79 -0
79 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +69 -4
185 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +23 -2
@@ -47,6 +47,7 @@#include "fpga/ipsec.h" #include "en_accel/ipsec_rxtx.h" #include "en_accel/tls_rxtx.h" +#include "en_accel/nvmeotcp.h" #include "lib/clock.h" #include "en/xdp.h" #include "en/xsk/rx.h"
@@ -617,16 +618,26 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq) ci =mlx5_wq_cyc_ctr2ix (&sq->wq, sqcc); wi = &sq->db.wqe_info[ci]; sqcc += wi->num_wqebbs; -#ifdef CONFIG_MLX5_EN_TLS switch (wi->wqe_type) { +#ifdef CONFIG_MLX5_EN_TLS case MLX5E_ICOSQ_WQE_SET_PSV_TLS :mlx5e_ktls_handle_ctx_completion (wi);break ;case MLX5E_ICOSQ_WQE_GET_PSV_TLS :mlx5e_ktls_handle_get_psv_completion (wi, sq);break ; - }#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +case MLX5E_ICOSQ_WQE_UMR_NVME_TCP : +break ; +case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE : +mlx5e_nvmeotcp_ddp_inv_done (wi); +break ; +case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP : +mlx5e_nvmeotcp_ctx_comp (wi); +break ; +#endif + } } sq->cc = sqcc; }
@@ -695,6 +706,16 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)case MLX5E_ICOSQ_WQE_GET_PSV_TLS :mlx5e_ktls_handle_get_psv_completion (wi, sq);break ; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +case MLX5E_ICOSQ_WQE_UMR_NVME_TCP : +break ; +case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE : +mlx5e_nvmeotcp_ddp_inv_done (wi); +break ; +case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP : +mlx5e_nvmeotcp_ctx_comp (wi); +break ;#endif default : netdev_WARN_ONCE(cq->channel->netdev,
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +26 -0
@@ -34,6 +34,7 @@#include "en.h" #include "en_accel/tls.h" #include "en_accel/en_accel.h" +#include "en_accel/nvmeotcp.h" static unsigned int stats_grps_num (struct mlx5e_priv *priv) {
@@ -189,6 +190,14 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_ok) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_skip) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) }, +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init_fail) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_teardown) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) }, #endif { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
@@ -314,6 +323,14 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw) s->rx_tls_resync_res_ok += rq_stats->tls_resync_res_ok; s->rx_tls_resync_res_skip += rq_stats->tls_resync_res_skip; s->rx_tls_err += rq_stats->tls_err; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + s->rx_nvmeotcp_queue_init += rq_stats->nvmeotcp_queue_init; + s->rx_nvmeotcp_queue_init_fail += rq_stats->nvmeotcp_queue_init_fail; + s->rx_nvmeotcp_queue_teardown += rq_stats->nvmeotcp_queue_teardown; + s->rx_nvmeotcp_ddp_setup += rq_stats->nvmeotcp_ddp_setup; + s->rx_nvmeotcp_ddp_setup_fail += rq_stats->nvmeotcp_ddp_setup_fail; + s->rx_nvmeotcp_ddp_teardown += rq_stats->nvmeotcp_ddp_teardown;#endif s->ch_events += ch_stats->events; s->ch_poll += ch_stats->poll;
@@ -390,6 +407,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw) s->tx_tls_drop_no_sync_data += sq_stats->tls_drop_no_sync_data; s->tx_tls_drop_bypass_req += sq_stats->tls_drop_bypass_req;#endif + s->tx_cqes += sq_stats->cqes;/* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92657 */
@@ -1559,6 +1577,14 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_resync_res_skip) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) },#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init_fail) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_teardown) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) }, +#endif }; static const struct counter_desc sq_stats_desc[] = {
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +16 -0
@@ -175,6 +175,14 @@ struct mlx5e_sw_stats {u64 rx_congst_umr;u64 rx_arfs_err;u64 rx_recover; +#ifdef CONFIG_MLX5_EN_NVMEOTCP +u64 rx_nvmeotcp_queue_init; +u64 rx_nvmeotcp_queue_init_fail; +u64 rx_nvmeotcp_queue_teardown; +u64 rx_nvmeotcp_ddp_setup; +u64 rx_nvmeotcp_ddp_setup_fail; +u64 rx_nvmeotcp_ddp_teardown; +#endif u64 ch_events;u64 ch_poll;u64 ch_arm;
@@ -338,6 +346,14 @@ struct mlx5e_rq_stats {u64 tls_resync_res_skip;u64 tls_err;#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +u64 nvmeotcp_queue_init; +u64 nvmeotcp_queue_init_fail; +u64 nvmeotcp_queue_teardown; +u64 nvmeotcp_ddp_setup; +u64 nvmeotcp_ddp_setup_fail; +u64 nvmeotcp_ddp_teardown; +#endif };struct mlx5e_sq_stats {
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +6 -0
@@ -158,6 +158,9 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)* queueing more WQEs and overflowing the async ICOSQ. */ clear_bit (MLX5E_SQ_STATE_PENDING_XSK_TX , &c->async_icosq.state); +#ifdef CONFIG_MLX5_EN_NVMEOTCP +mlx5e_poll_ico_cq (&c->nvmeotcpsq.cq); +#endif busy |= INDIRECT_CALL_2(rq->post_wqes, mlx5e_post_rx_mpwqes,
@@ -196,6 +199,9 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)mlx5e_cq_arm (&rq->cq);mlx5e_cq_arm (&c->icosq.cq);mlx5e_cq_arm (&c->async_icosq.cq); +#ifdef CONFIG_MLX5_EN_NVMEOTCP +mlx5e_cq_arm (&c->nvmeotcpsq.cq); +#endif mlx5e_cq_arm (&c->xdpsq.cq);if (xsk_open) {
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
@@ -256,6 +256,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)return err; } +if (MLX5_CAP_GEN (dev, nvmeotcp)) { + err =mlx5_core_get_caps (dev,MLX5_CAP_DEV_NVMEOTCP ); +if (err) +return err; + } +return 0 ; }
include/linux/mlx5/device.h +8 -0
@@ -263,6 +263,7 @@ enum {enum {MLX5_MKEY_MASK_LEN =1ull <<0 ,MLX5_MKEY_MASK_PAGE_SIZE =1ull <<1 , +MLX5_MKEY_MASK_XLT_OCT_SIZE =1ull <<2 ,MLX5_MKEY_MASK_START_ADDR =1ull <<6 ,MLX5_MKEY_MASK_PD =1ull <<7 ,MLX5_MKEY_MASK_EN_RINVAL =1ull <<8 ,
@@ -1153,6 +1154,7 @@ enum mlx5_cap_type {MLX5_CAP_VDPA_EMULATION =0x13 ,MLX5_CAP_DEV_EVENT =0x14 ,MLX5_CAP_IPSEC , +MLX5_CAP_DEV_NVMEOTCP =0x19 ,/* NUM OF CAP Types */ MLX5_CAP_NUM };
@@ -1373,6 +1375,12 @@ enum mlx5_qcam_feature_groups {#define MLX5_CAP_IPSEC (mdev, cap)\ MLX5_GET(ipsec_cap, (mdev)->caps.hca_cur[MLX5_CAP_IPSEC], cap) +#define MLX5_CAP_DEV_NVMEOTCP (mdev, cap)\ + MLX5_GET(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap) + +#define MLX5_CAP64_NVMEOTCP (mdev, cap)\ + MLX5_GET64(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap) +enum {MLX5_CMD_STAT_OK =0x0 ,MLX5_CMD_STAT_INT_ERR =0x1 ,
include/linux/mlx5/mlx5_ifc.h +117 -4
184 lines not on this page · raw message
include/linux/mlx5/qp.h +1 -0
@@ -220,6 +220,7 @@ struct mlx5_wqe_ctrl_seg {#define MLX5_WQE_CTRL_OPCODE_MASK 0xff#define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00#define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8 +#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8enum {MLX5_ETH_WQE_L3_INNER_CSUM =1 <<4 ,
Signature · 1 line
2.24.1
kernel test robot
build botRe: [PATCH net-next RFC v1 09/10] net/mlx5e: Add NVMEoTCP offload
reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…
Hi Boris,
[FYI, it's a private test report for your RFC patch.] [auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1 config: xtensa-allyesconfig (attached as .config) compiler: xtensa-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/990254a46f72010830cdf80ed80ed506f268e759
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 git checkout 990254a46f72010830cdf80ed80ed506f268e759
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=xtensa
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
In file included from include/linux/bits.h:6,
from include/linux/bitops.h:5,
from include/linux/kernel.h:12,
from include/linux/list.h:9,
from include/linux/timer.h:5,
from include/linux/netdevice.h:24,
from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:4:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]
7 | #define BIT(nr) (UL(1) << (nr))
| ^~
include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
10680 | MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
| ^~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:68:6: error: redefinition of 'mlx5e_accel_fs_del_sk'
68 | void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
| ^~~~~~~~~~~~~~~~~~~~~
In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:23:20: note: previous definition of 'mlx5e_accel_fs_del_sk' was here
23 | static inline void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule) {}
| ^~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:73:26: error: redefinition of 'mlx5e_accel_fs_add_sk'
73 | struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
| ^~~~~~~~~~~~~~~~~~~~~
In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:19:40: note: previous definition of 'mlx5e_accel_fs_add_sk' was here
19 | static inline struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
| ^~~~~~~~~~~~~~~~~~~~~
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_add_sk':
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:88:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
88 | fs_tcp = priv->fs.accel_tcp;
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_add_default_rule':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:160:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
160 | fs_tcp = priv->fs.accel_tcp;
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_create_table':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:269:41: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
269 | struct mlx5e_flow_table *ft = &priv->fs.accel_tcp->tables[type];
| ^
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:276:18: error: 'MLX5E_ACCEL_FS_TCP_FT_LEVEL' undeclared (first use in this function); did you mean 'MLX5E_ACCEL_FS_ESP_FT_LEVEL'?
276 | ft_attr.level = MLX5E_ACCEL_FS_TCP_FT_LEVEL;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
| MLX5E_ACCEL_FS_ESP_FT_LEVEL
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:276:18: note: each undeclared identifier is reported only once for each function it appears in
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_enable':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:328:21: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
328 | dest.ft = priv->fs.accel_tcp->tables[i].t;
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:339:24: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
339 | refcount_set(&priv->fs.accel_tcp->ref_count, 1);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'accel_fs_tcp_destroy_table':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:347:19: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
347 | fs_tcp = priv->fs.accel_tcp;
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: At top level:
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:356:6: error: redefinition of 'mlx5e_accel_fs_tcp_destroy'
356 | void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:18:20: note: previous definition of 'mlx5e_accel_fs_tcp_destroy' was here
18 | static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_tcp_destroy':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:360:15: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
360 | if (!priv->fs.accel_tcp)
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:363:38: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
363 | if (!refcount_dec_and_test(&priv->fs.accel_tcp->ref_count))
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:371:16: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
371 | kfree(priv->fs.accel_tcp);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:372:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
372 | priv->fs.accel_tcp = NULL;
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: At top level:
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:375:5: error: redefinition of 'mlx5e_accel_fs_tcp_create'
375 | int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:5:
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h:17:19: note: previous definition of 'mlx5e_accel_fs_tcp_create' was here
17 | static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
| ^~~~~~~~~~~~~~~~~~~~~~~~~
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c: In function 'mlx5e_accel_fs_tcp_create':
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:382:14: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
382 | if (priv->fs.accel_tcp) {
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:383:25: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
383 | refcount_inc(&priv->fs.accel_tcp->ref_count);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:387:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
387 | priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:387:47: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
387 | priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:388:15: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
388 | if (!priv->fs.accel_tcp)
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:407:16: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
407 | kfree(priv->fs.accel_tcp);
| ^
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c:408:10: error: 'struct mlx5e_flow_steering' has no member named 'accel_tcp'
408 | priv->fs.accel_tcp = NULL;
| ^--
In file included from include/linux/bits.h:6,
from include/linux/bitops.h:5,
from include/linux/kernel.h:12,
from include/linux/list.h:9,
from include/linux/timer.h:5,
from include/linux/netdevice.h:24,
from drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c:4:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]
7 | #define BIT(nr) (UL(1) << (nr))
| ^~
include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
10680 | MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
| ^~~
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c: In function 'mlx5e_nvmeotcp_queue_init':
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c:613:7: warning: 'stats' is used uninitialized in this function [-Wuninitialized]
613 | stats->nvmeotcp_queue_init_fail++;
| ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~vim +/mlx5e_accel_fs_del_sk +68 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
5229a96e59ec324 Tariq Toukan 2020-06-16 67
5229a96e59ec324 Tariq Toukan 2020-06-16 @68 void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
5229a96e59ec324 Tariq Toukan 2020-06-16 69 {
5229a96e59ec324 Tariq Toukan 2020-06-16 70 mlx5_del_flow_rules(rule);
5229a96e59ec324 Tariq Toukan 2020-06-16 71 }
5229a96e59ec324 Tariq Toukan 2020-06-16 72
5229a96e59ec324 Tariq Toukan 2020-06-16 @73 struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
5229a96e59ec324 Tariq Toukan 2020-06-16 74 struct sock *sk, u32 tirn,
5229a96e59ec324 Tariq Toukan 2020-06-16 75 uint32_t flow_tag)
5229a96e59ec324 Tariq Toukan 2020-06-16 76 {
5229a96e59ec324 Tariq Toukan 2020-06-16 77 struct mlx5_flow_destination dest = {};
5229a96e59ec324 Tariq Toukan 2020-06-16 78 struct mlx5e_flow_table *ft = NULL;
5229a96e59ec324 Tariq Toukan 2020-06-16 79 struct mlx5e_accel_fs_tcp *fs_tcp;
5229a96e59ec324 Tariq Toukan 2020-06-16 80 MLX5_DECLARE_FLOW_ACT(flow_act);
5229a96e59ec324 Tariq Toukan 2020-06-16 81 struct mlx5_flow_handle *flow;
5229a96e59ec324 Tariq Toukan 2020-06-16 82 struct mlx5_flow_spec *spec;
5229a96e59ec324 Tariq Toukan 2020-06-16 83
5229a96e59ec324 Tariq Toukan 2020-06-16 84 spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
5229a96e59ec324 Tariq Toukan 2020-06-16 85 if (!spec)
5229a96e59ec324 Tariq Toukan 2020-06-16 86 return ERR_PTR(-ENOMEM);
5229a96e59ec324 Tariq Toukan 2020-06-16 87
5229a96e59ec324 Tariq Toukan 2020-06-16 @88 fs_tcp = priv->fs.accel_tcp;
5229a96e59ec324 Tariq Toukan 2020-06-16 89
5229a96e59ec324 Tariq Toukan 2020-06-16 90 spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
5229a96e59ec324 Tariq Toukan 2020-06-16 91
5229a96e59ec324 Tariq Toukan 2020-06-16 92 switch (sk->sk_family) {
5229a96e59ec324 Tariq Toukan 2020-06-16 93 case AF_INET:
5229a96e59ec324 Tariq Toukan 2020-06-16 94 accel_fs_tcp_set_ipv4_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16 95 ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16 96 mlx5e_dbg(HW, priv, "%s flow is %pI4:%d -> %pI4:%d\n", __func__,
5229a96e59ec324 Tariq Toukan 2020-06-16 97 &inet_sk(sk)->inet_rcv_saddr,
5229a96e59ec324 Tariq Toukan 2020-06-16 98 inet_sk(sk)->inet_sport,
5229a96e59ec324 Tariq Toukan 2020-06-16 99 &inet_sk(sk)->inet_daddr,
5229a96e59ec324 Tariq Toukan 2020-06-16 100 inet_sk(sk)->inet_dport);
5229a96e59ec324 Tariq Toukan 2020-06-16 101 break;
5229a96e59ec324 Tariq Toukan 2020-06-16 102 #if IS_ENABLED(CONFIG_IPV6)
5229a96e59ec324 Tariq Toukan 2020-06-16 103 case AF_INET6:
5229a96e59ec324 Tariq Toukan 2020-06-16 104 if (!sk->sk_ipv6only &&
5229a96e59ec324 Tariq Toukan 2020-06-16 105 ipv6_addr_type(&sk->sk_v6_daddr) == IPV6_ADDR_MAPPED) {
5229a96e59ec324 Tariq Toukan 2020-06-16 106 accel_fs_tcp_set_ipv4_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16 107 ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16 108 } else {
5229a96e59ec324 Tariq Toukan 2020-06-16 109 accel_fs_tcp_set_ipv6_flow(spec, sk);
5229a96e59ec324 Tariq Toukan 2020-06-16 110 ft = &fs_tcp->tables[ACCEL_FS_IPV6_TCP];
5229a96e59ec324 Tariq Toukan 2020-06-16 111 }
5229a96e59ec324 Tariq Toukan 2020-06-16 112 break;
5229a96e59ec324 Tariq Toukan 2020-06-16 113 #endif
5229a96e59ec324 Tariq Toukan 2020-06-16 114 default:
5229a96e59ec324 Tariq Toukan 2020-06-16 115 break;
5229a96e59ec324 Tariq Toukan 2020-06-16 116 }
5229a96e59ec324 Tariq Toukan 2020-06-16 117
5229a96e59ec324 Tariq Toukan 2020-06-16 118 if (!ft) {
5229a96e59ec324 Tariq Toukan 2020-06-16 119 flow = ERR_PTR(-EINVAL);
5229a96e59ec324 Tariq Toukan 2020-06-16 120 goto out;
5229a96e59ec324 Tariq Toukan 2020-06-16 121 }
5229a96e59ec324 Tariq Toukan 2020-06-16 122
5229a96e59ec324 Tariq Toukan 2020-06-16 123 MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
5229a96e59ec324 Tariq Toukan 2020-06-16 124 outer_headers.tcp_dport);
5229a96e59ec324 Tariq Toukan 2020-06-16 125 MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
5229a96e59ec324 Tariq Toukan 2020-06-16 126 outer_headers.tcp_sport);
5229a96e59ec324 Tariq Toukan 2020-06-16 127 MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_dport,
5229a96e59ec324 Tariq Toukan 2020-06-16 128 ntohs(inet_sk(sk)->inet_sport));
5229a96e59ec324 Tariq Toukan 2020-06-16 129 MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_sport,
5229a96e59ec324 Tariq Toukan 2020-06-16 130 ntohs(inet_sk(sk)->inet_dport));
5229a96e59ec324 Tariq Toukan 2020-06-16 131
5229a96e59ec324 Tariq Toukan 2020-06-16 132 dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
5229a96e59ec324 Tariq Toukan 2020-06-16 133 dest.tir_num = tirn;
5229a96e59ec324 Tariq Toukan 2020-06-16 134 if (flow_tag != MLX5_FS_DEFAULT_FLOW_TAG) {
5229a96e59ec324 Tariq Toukan 2020-06-16 135 spec->flow_context.flow_tag = flow_tag;
5229a96e59ec324 Tariq Toukan 2020-06-16 136 spec->flow_context.flags = FLOW_CONTEXT_HAS_TAG;
5229a96e59ec324 Tariq Toukan 2020-06-16 137 }
5229a96e59ec324 Tariq Toukan 2020-06-16 138
5229a96e59ec324 Tariq Toukan 2020-06-16 139 flow = mlx5_add_flow_rules(ft->t, spec, &flow_act, &dest, 1);
5229a96e59ec324 Tariq Toukan 2020-06-16 140
5229a96e59ec324 Tariq Toukan 2020-06-16 141 if (IS_ERR(flow))
5229a96e59ec324 Tariq Toukan 2020-06-16 142 netdev_err(priv->netdev, "mlx5_add_flow_rules() failed, flow is %ld\n",
5229a96e59ec324 Tariq Toukan 2020-06-16 143 PTR_ERR(flow));
5229a96e59ec324 Tariq Toukan 2020-06-16 144
5229a96e59ec324 Tariq Toukan 2020-06-16 145 out:
5229a96e59ec324 Tariq Toukan 2020-06-16 146 kvfree(spec);
5229a96e59ec324 Tariq Toukan 2020-06-16 147 return flow;
5229a96e59ec324 Tariq Toukan 2020-06-16 148 }
5229a96e59ec324 Tariq Toukan 2020-06-16 149 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [PATCH net-next RFC v1 09/10] net/mlx5e: Add NVMEoTCP offload
In reply to Boris Pismennyreproduce (this is a W=1 build):
Hi Boris,
[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1 config: i386-allyesconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/990254a46f72010830cdf80ed80ed506f268e759 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 git checkout 990254a46f72010830cdf80ed80ed506f268e759 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
In file included from include/linux/bits.h:6,
from include/linux/ratelimit_types.h:5,
from include/linux/ratelimit.h:5,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/rdma/ib_verbs.h:16,
from drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c:33:
include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]
7 | #define BIT(nr) (UL(1) << (nr))
| ^~
include/linux/mlx5/mlx5_ifc.h:10680:64: note: in expansion of macro 'BIT'
10680 | MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT(0x21),
| ^~~
drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c: In function 'mlx5i_grp_sw_update_stats':
>> drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c:143:1: warning: the frame size of 1048 bytes is larger than 1024 bytes [-Wframe-larger-than=]
143 | }
| ^vim +143 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
48935bbb7ae8bd4 drivers/net/ethernet/mellanox/mlx5/core/ipoib.c Saeed Mahameed 2017-04-13 117
fbb66ad5dcbebc1 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Wei Yongjun 2018-09-05 118 static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv)
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 119 {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 120 struct mlx5e_sw_stats s = { 0 };
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 121 int i, j;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 122
694826e366349d5 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Tariq Toukan 2019-07-14 123 for (i = 0; i < priv->max_nch; i++) {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 124 struct mlx5e_channel_stats *channel_stats;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 125 struct mlx5e_rq_stats *rq_stats;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 126
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 127 channel_stats = &priv->channel_stats[i];
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 128 rq_stats = &channel_stats->rq;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 129
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 130 s.rx_packets += rq_stats->packets;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 131 s.rx_bytes += rq_stats->bytes;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 132
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 133 for (j = 0; j < priv->max_opened_tc; j++) {
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 134 struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 135
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 136 s.tx_packets += sq_stats->packets;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 137 s.tx_bytes += sq_stats->bytes;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 138 s.tx_queue_dropped += sq_stats->dropped;
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 139 }
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 140 }
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 141
c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 142 memcpy(&priv->stats.sw, &s, sizeof(s));c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 @143 }c57d2358ff0dfa0 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c Feras Daoud 2018-09-02 144
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 10/10 net/mlx5e: NVMEoTCP, data-path for DDP offload
In reply to Boris PismennyNVMEoTCP direct data placement constructs an SKB from each CQE, while pointing at NVME buffers.
This enables the offload, as the NVMe-TCP layer will skip the copy when src == dst.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 1 + .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 256 ++++++++++++++++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.h | 25 ++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 51 +++- .../ethernet/mellanox/mlx5/core/en_stats.c | 12 + .../ethernet/mellanox/mlx5/core/en_stats.h | 8 + include/linux/mlx5/device.h | 30 +- 10 files changed, 379 insertions(+), 8 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -85,4 +85,4 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o steering/dr_cmd.o steering/dr_fw.o \ steering/dr_action.o steering/fs_dr.o -mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -536,6 +536,7 @@ struct mlx5e_rq;typedef void (*mlx5e_fp_handle_rx_cqe )(struct mlx5e_rq *,struct mlx5_cqe64 *);typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe_mpwrq )(struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, +struct mlx5_cqe64 *cqe,u16 cqe_bcnt,u32 head_offset,u32 page_idx);typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe)(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +1 -0
@@ -25,6 +25,7 @@ static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, void *data,struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear (struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, +struct mlx5_cqe64 *cqe,u16 cqe_bcnt,u32 head_offset,u32 page_idx)
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +1 -0
@@ -11,6 +11,7 @@struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear (struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, +struct mlx5_cqe64 *cqe,u16 cqe_bcnt,u32 head_offset,u32 page_idx);
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c new file +256 -0
256 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +// Copyright (c) 2020 Mellanox Technologies. + +#ifndef __MLX5E_NVMEOTCP_RXTX_H__ +#define __MLX5E_NVMEOTCP_RXTX_H__ + +#ifdef CONFIG_MLX5_EN_NVMEOTCP + +#include <linux/skbuff.h> +#include "en.h" + +struct sk_buff * +mlx5e_nvmeotcp_handle_rx_skb (struct net_device *netdev,struct sk_buff *skb, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt,bool linear); + +int mlx5_nvmeotcp_get_headlen (struct mlx5_cqe64 *cqe,u32 cqe_bcnt); +#else +int mlx5_nvmeotcp_get_headlen (struct mlx5_cqe64 *cqe,u32 cqe_bcnt) {return cqe_bcnt; } +struct sk_buff * +mlx5e_nvmeotcp_handle_rx_skb (struct net_device *netdev,struct sk_buff *skb, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt,bool linear) {return skb; } + +#endif /* CONFIG_MLX5_EN_NVMEOTCP */ + +#endif /* __MLX5E_NVMEOTCP_RXTX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +45 -6
143 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +12 -0
@@ -198,6 +198,10 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_drop) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_resync) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_bytes) }, #endif { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
@@ -331,6 +335,10 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw) s->rx_nvmeotcp_ddp_setup += rq_stats->nvmeotcp_ddp_setup; s->rx_nvmeotcp_ddp_setup_fail += rq_stats->nvmeotcp_ddp_setup_fail; s->rx_nvmeotcp_ddp_teardown += rq_stats->nvmeotcp_ddp_teardown; + s->rx_nvmeotcp_drop += rq_stats->nvmeotcp_drop; + s->rx_nvmeotcp_resync += rq_stats->nvmeotcp_resync; + s->rx_nvmeotcp_offload_packets += rq_stats->nvmeotcp_offload_packets; + s->rx_nvmeotcp_offload_bytes += rq_stats->nvmeotcp_offload_bytes;#endif s->ch_events += ch_stats->events; s->ch_poll += ch_stats->poll;
@@ -1584,6 +1592,10 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_drop) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_resync) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_bytes) }, #endif };
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +8 -0
@@ -182,6 +182,10 @@ struct mlx5e_sw_stats {u64 rx_nvmeotcp_ddp_setup;u64 rx_nvmeotcp_ddp_setup_fail;u64 rx_nvmeotcp_ddp_teardown; +u64 rx_nvmeotcp_drop; +u64 rx_nvmeotcp_resync; +u64 rx_nvmeotcp_offload_packets; +u64 rx_nvmeotcp_offload_bytes;#endif u64 ch_events;u64 ch_poll;
@@ -353,6 +357,10 @@ struct mlx5e_rq_stats {u64 nvmeotcp_ddp_setup;u64 nvmeotcp_ddp_setup_fail;u64 nvmeotcp_ddp_teardown; +u64 nvmeotcp_drop; +u64 nvmeotcp_resync; +u64 nvmeotcp_offload_packets; +u64 nvmeotcp_offload_bytes;#endif };
include/linux/mlx5/device.h +29 -1
@@ -779,7 +779,7 @@ struct mlx5_err_cqe {struct mlx5_cqe64 {u8 tls_outer_l3_tunneled; -u8 rsvd0; +u8 nvmetcp;__be16 wqe_id;u8 lro_tcppsh_abort_dupack;u8 lro_min_ttl;
@@ -812,6 +812,19 @@ struct mlx5_cqe64 {u8 op_own; }; +struct mlx5e_cqe128 { +__be16 cclen; +__be16 hlen; +union { +__be32 resync_tcp_sn; +__be32 ccoff; + }; +__be16 ccid; +__be16 rsvd8; +u8 rsvd12[52 ]; +struct mlx5_cqe64 cqe64; +}; + struct mlx5_mini_cqe8 { union { __be32 rx_hash_result;
@@ -842,6 +855,21 @@ enum {#define MLX5_MINI_CQE_ARRAY_SIZE 8 +static inline bool cqe_is_nvmeotcp_resync (struct mlx5_cqe64 *cqe) +{ +return ((cqe->nvmetcp >>6 ) &0x1 ); +} + +static inline bool cqe_is_nvmeotcp_crcvalid (struct mlx5_cqe64 *cqe) +{ +return ((cqe->nvmetcp >>5 ) &0x1 ); +} + +static inline bool cqe_is_nvmeotcp_zc (struct mlx5_cqe64 *cqe) +{ +return ((cqe->nvmetcp >>4 ) &0x1 ); +} +static inline u8 mlx5_get_cqe_format (struct mlx5_cqe64 *cqe) {return (cqe->op_own >>2 ) &0x3 ;
Signature · 1 line
2.24.1
kernel test robot
build botRe: [PATCH net-next RFC v1 10/10] net/mlx5e: NVMEoTCP, data-path for DDP offload
reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…
Hi Boris,
[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 879456bedbe54f2d38b15c21dc5e3c30232b53e1 config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/26f3c706eccd3eb38acd731a4bfd2583077749bb
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Boris-Pismenny/nvme-tcp-receive-offloads/20201001-002308 git checkout 26f3c706eccd3eb38acd731a4bfd2583077749bb
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
In file included from drivers/net/ethernet/mellanox/mlx5/core/en_rx.c:51:
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h:18:5: warning: no previous prototype for 'mlx5_nvmeotcp_get_headlen' [-Wmissing-prototypes]
18 | int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { return cqe_bcnt; }
| ^~~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h:20:1: warning: no previous prototype for 'mlx5e_nvmeotcp_handle_rx_skb' [-Wmissing-prototypes] 20 | mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
vim +/mlx5_nvmeotcp_get_headlen +18 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
11
12 struct sk_buff*
13 mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
14 struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear);
15
16 int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt);
17 #else
> 18 int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) { return cqe_bcnt; }
19 struct sk_buff*
> 20 mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
21 struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear) { return skb; }
22 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Sagi Grimberg
maintainerQuoted 85 lines · Boris Pismenny
This series adds support for nvme-tcp receive offloads which do not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers
2. CRC verification for received PDUThe series implements these as a generic offload infrastructure for storage protocols, which we call TCP Direct Data Placement (
TCP_DDP) and TCP DDP CRC, respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC. Future implementations can reuse the same infrastructure for other protcols such as iSCSI.Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (seenet/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.htmlInitialization and teardown:
=========================================
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by callingnvme_tcp_offload_socketwith the tcp socket of thenvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usually with no offload. If it succeeds thennvme_tcp_setup_ddpandnvme_tcp_teardown_ddpmay be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of nvme-tcp in any way besides adding the option to call the above mentioned NDO operations.For the admin queue, nvme-tcp does not initialize the offload. Instead, nvme-tcp calls the driver to configure limits for the controller, such as
max_hw_sectorsandmax_segments; these must be limited to accomodate potential HW resource limits, and to improve performance.If some error occured, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.
Copy offload works as follows:
=========================================
The nvme-tcp layer calls the NIC drive to map block layer buffers to ccid usingnvme_tcp_setup_ddpbefore sending the read request. When the repsonse is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer; this SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once nvme-tcp attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function (memcpy_to_page):
if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the nvme-tcp layer releases the NIC HW context be callingnvme_tcp_teardown_ddpwhich asynchronously unmaps the buffers from NIC HW.As the last change is to a sensative function, we are careful to place it under
static_keywhich is only enabled when this functionality is actually used for nvme-tcp copy offload.Asynchronous completion:
=========================================
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance criticial, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).An alternative approach is to move all the functions related to coping from SKBs to the block layer buffers inside the nvme-tcp code - about 200 LOC.
CRC offload works as follows:
=========================================
After offload is initialized, we use the SKB'sddp_crcbit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (ddp_crc!= 1), then software must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.SKB changes:
=========================================
The CRC offload requires an additional bit in the SKB, which is useful for preventing the coalescing of SKB with different crc offload values. This bit is similar in concept to the "decrypted" bit.
Quoted 8 lines
Performance:
=========================================
The expected performance gain from this offload varies with the block size. We perform a CPU cycles breakdown of the copy/CRC operations in nvme-tcp fio random read workloads:
For 4K blocks we see up to 11% improvement for a 100% read fio workload, while for 128K blocks we see upto 52%. If we run nvme-tcp, and skip these operations, then we observe a gain of about 1.1x and 2x respectively.
Nice!
Quoted 33 lines
Resynchronization:
=========================================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the driver, regarding a possible location of a PDU header. Followed by a response from the nvme-tcp driver.This flow is rare, and it should happen only after packet loss or reordering events that involve nvme-tcp PDU headers.
The patches are organized as follows:
=========================================
Patch 1 theiov_iterchange to skip copy if (src == dst)
Patches 2-3 the infrastructure for all TCP DDP
and TCP DDP CRC offloads, respectively.Patch 4 exposes the get_netdev_for_sock function from TLS Patch 5 NVMe-TCP changes to call NIC driver on queue init/teardown Patches 6 NVMe-TCP changes to call NIC driver on IO operationsetup/teardown, and support async completions.
Patches 7 NVMe-TCP changes to support CRC offload on receive.
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patches 8 NVMe-TCP handling of netdev events: stop the offload if
netdev is going down
Patches 9-10 implement support for NVMe-TCP copy and CRC offload in
the mlx5 NIC driverTesting:
=========================================
This series was tested using fio with various configurations of IO sizes, depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets.
Quoted 5 lines
Future work:
=========================================
A follow-up series will introduce support for transmit side CRC. Then, we will work on adding support for TLS in NVMe-TCP and combining the two offloads.
Boris, Or and Yoray
Thanks for submitting this work. Overall this looks good to me.
The model here is not messy at all which is not trivial when it comes to tcp offloads.
Gave you comments in the patches themselves but overall this looks good!
Would love to see TLS work from you moving forward.