nvme-tcp receive offloads
Subsystems (14)
v12 cover nvme-tcp receive offloads
Hi,
The next iteration of our nvme-tcp receive offload series. This submission was rebased on top of net-next, and on top of the NVMe-TCP TLS implementation.
https://lore.kernel.org/all/[email protected]
https://lore.kernel.org/r/[email protected]
The main change was adding a patch to check for TLS support before enabling NVMe-TCP offload.
The changes are also available through git:
Repo: https://github.com/aaptel/linux.git branch nvme-rx-offload-v12
Web: https://github.com/aaptel/linux/tree/nvme-rx-offload-v12
The NVMe-TCP offload was presented in netdev 0x16 (video now available):
- https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains
- https://youtu.be/W74TR-SNgi4
From: Aurelien Aptel <[email protected]>
From: Shai Malin <[email protected]>
From: Boris Pismenny <[email protected]>
From: Or Gerlitz <[email protected]>
From: Yoray Zack <[email protected]>
From: Max Gurtovoy <[email protected]>
=========================================
This series adds support for NVMe-TCP receive offloads. The method here does not mandate the offload of the network stack to the device. Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers.
2. CRC calculation and verification for received PDU.
The series implements these as a generic offload infrastructure for storage protocols, which calls TCP Direct Data Placement and TCP Offload CRC respectively. We use this infrastructure to implement NVMe-TCP offload for copy and CRC.
Future implementations can reuse the same infrastructure for other protocols such as iSCSI.
Note:
These offloads are similar in nature to the packet-based NIC TLS offloads, which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html
Queue Level
===========
The offload for IO queues is initialized after the handshake of the NVMe-TCP protocol is finished by calling nvme_tcp_offload_socket with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usual with no offload. If it succeeds then nvme_tcp_setup_ddp and nvme_tcp_teardown_ddp may be called to perform copy offload, and crc offload will be used. This initialization does not change the normal operation of NVMe-TCP in any way besides adding the option to call the above mentioned NDO operations.
For the admin queue, NVMe-TCP does not initialize the offload. Instead, NVMe-TCP calls the driver to configure limits for the controller, such as max_hw_sectors and max_segments, these must be limited to accommodate potential HW resource limits, and to improve performance.
If some error occurs, and the IO queue must be closed or reconnected, then offload is teardown and initialized again. Additionally, we handle netdev down events via the existing error recovery flow.
IO Level
========
The NVMe-TCP layer calls the NIC driver to map block layer buffers to CID using nvme_tcp_setup_ddp before sending the read request. When the response is received, then the NIC HW will write the PDU payload directly into the designated buffer, and build an SKB such that it points into the destination buffer. This SKB represents the entire packet received on the wire, but it points to the block layer buffers. Once NVMe-TCP attempts to copy data from this SKB to the block layer buffer it can skip the copy by checking in the copying function: if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the NVMe-TCP layer releases the NIC HW context by calling nvme_tcp_teardown_ddp which asynchronously unmaps the buffers from NIC HW.
The NIC must release its mapping between command IDs and the target buffers. This mapping is released when NVMe-TCP calls the NIC
driver (nvme_tcp_offload_socket).
As completing IOs is performance critical, we introduce asynchronous completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later call NVMe-TCP to complete the IO (nvme_tcp_ddp_teardown_done).
On the IO level, and in order to use the offload only when a clear performance improvement is expected, the offload is used only for IOs which are bigger than io_threshold.
SKB
===
The DDP (zero-copy) and CRC offloads require two additional bits in the SKB. The ddp bit is useful to prevent condensing of SKBs which are targeted for zero-copy. The crc bit is useful to prevent GRO coalescing SKBs with different offload values. This bit is similar in concept to the "decrypted" bit.
After offload is initialized, we use the SKB's crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's payload". The bit is set to zero if there was an error, or if HW skipped offload for some reason. If *any* SKB in a PDU has (crc != 1), then the calling driver must compute the CRC, and check it. We perform this check, and accompanying software fallback at the end of the processing of a received PDU.
Resynchronization flow
======================
The resynchronization flow is performed to reset the hardware tracking of NVMe-TCP PDUs within the TCP stream. The flow consists of a request from the hardware proxied by the driver, regarding a possible location of a PDU header. Followed by a response from the NVMe-TCP driver.
This flow is rare, and it should happen only after packet loss or reordering events that involve NVMe-TCP PDU headers.
CID Mapping
===========
ConnectX-7 assumes linear CID (0...N-1 for queue of size N) where the Linux NVMe driver uses part of the 16 bit CCID for generation counter.
To address that, we use the existing quirk in the NVMe layer when the HW driver advertises that they don't support the full 16 bit CCID range.
Enablement on ConnectX-7
========================
By default, NVMeTCP offload is disabled in the mlx driver and in the nvme-tcp host. In order to enable it:
# Disable CQE compression (specific for ConnectX) ethtool --set-priv-flags <device> rx_cqe_compress off
# Enable the ULP-DDP
ethtool --ulp-ddp <device> nvme-tcp-ddp on nvme-tcp-ddgst-rx-offload on
# Enable ULP offload in nvme-tcp
modprobe nvme-tcp ulp_offload=1
Following the device ULP-DDP enablement, all the IO queues/sockets which are running on the device are offloaded.
This feature requires a patched ethtool that can be obtained from
Web: https://github.com/aaptel/ethtool/tree/ulp-ddp
Git: https://github.com/aaptel/ethtool.git
Branch: ulp-ddp
Performance
===========
With this implementation, using the ConnectX-7 NIC, we were able to demonstrate the following CPU utilization improvement:
Without data digest:
For 64K queued read IOs – up to 32% improvement in the BW/IOPS (111 Gbps vs. 84 Gbps). For 512K queued read IOs – up to 55% improvement in the BW/IOPS (148 Gbps vs. 98 Gbps).
With data digest:
For 64K queued read IOs – up to 107% improvement in the BW/IOPS (111 Gbps vs. 53 Gbps). For 512K queued read IOs – up to 138% improvement in the BW/IOPS (146 Gbps vs. 61 Gbps).
With small IOs we are not expecting that the offload will show a performance gain.
The test configuration:
- fio command: qd=128, jobs=8.
- Server: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 160 cores.
Patches
=======
Patch 1: Introduce the infrastructure for all ULP DDP and ULP DDP CRC offloads. Patch 2: Add ethtool string sets for ULP DDP capability names and stats names. Patch 3: Add ethtool operations to get/set ULP DDP capabilities and statistics. Patch 4: Documentation of ULP DDP ethtool netlink messages. Patch 5: The iov_iter change to skip copy if (src == dst). Patch 6: Export the get_netdev_for_sock function from TLS to generic location. Patch 7: NVMe-TCP changes to call NIC driver on queue init/teardown and resync. Patch 8: NVMe-TCP changes to call NIC driver on IO operation
setup/teardown, and support async completions.
Patch 9: NVMe-TCP changes to support CRC offload on receive
Also, this patch moves CRC calculation to the end of PDU in case offload requires software fallback.
Patch 10: NVMe-TCP handling of netdev events: stop the offload if netdev is
going down.
Patch 11: Add module parameter to the NVMe-TCP control the enable ULP offload.
Patch 12: Check if netdev supports combining TLS and offload before
enabling offloading.
Patch 13: Documentation of ULP DDP offloads.
The rest of the series is the mlx5 implementation of the offload.
Testing
=======
This series was tested on ConnectX-7 HW using various configurations of IO sizes, queue depths, MTUs, and with both the SPDK and kernel NVMe-TCP targets.
Compatibility
=============
* The offload works with bare-metal or SRIOV.
* The HW can support up to 64K connections per device (assuming no other HW accelerations are used). In this series, we will introduce the support for up to 4k connections, and we have plans to increase it.
* In the current HW implementation, the combination of NVMeTCP offload with TLS is not supported. In the future, if it will be implemented, the impact on the NVMe/TCP layer will be minimal.
* The NVMeTCP offload ConnectX 7 HW can support tunneling, but we don't see the need for this feature yet.
* NVMe poll queues are not in the scope of this series.
Future Work
===========
* NVMeTCP transmit offload.
* NVMeTCP offloads incremental features.
Changes since v11:
=================
- Rebase on top of NVMe-TCP kTLS offload.
- Add tls support bit in struct ulp_ddp_limits.
- Simplify logic in NVMe-TCP queue init.
- Use new page pool in mlx5 driver.
Changes since v10:
=================
- Pass extack to drivers for better error reporting in the .set_caps callback (Jakub).
- netlink: use new callbacks, existing macros, padding, fix size add notifications, update specs (Jakub).
Changes since v9:
=================
- Add missing crc checks in tcp_try_coalesce() (Paolo).
- Add missing ifdef guard for socket ops (Paolo).
- Remove verbose netlink format for statistics (Jakub).
- Use regular attributes for statistics (Jakub).
- Expose and document individual stats to uAPI (Jakub).
- Move ethtool ops for caps&stats to netdev_ops->ulp_ddp_ops (Jakub).
Changes since v8:
=================
- Make stats stringset global instead of per-device (Jakub).
- Remove per-queue stats (Jakub).
- Rename ETH_SS_ULP_DDP stringset to ETH_SS_ULP_DDP_CAPS.
- Update & fix kdoc comments.
- Use 80 columns limit for nvme code.
Changes since v7:
=================
- Remove ULP DDP netdev->feature bit (Jakub).
- Expose ULP DDP capabilities to userspace via ethtool netlink messages (Jakub).
- Move ULP DDP stats to dedicated stats group (Jakub).
- Add ethtool_ops operations for setting capabilities and getting stats (Jakub).
- Move ulp_ddp_netdev_ops into net_device_ops (Jakub).
- Use union for protocol-specific struct instances (Jakub).
- Fold netdev_sk_get_lowest_dev() into get_netdev_for_sock (Christoph).
- Rename memcpy skip patch to something more obvious (Christoph).
- Move capabilities from ulp_ddp.h to ulp_ddp_caps.h.
- Add "Compatibility" section on the cover letter (Sagi).
Changes since v6:
=================
- Moved IS_ULP_{DDP,CRC} macros to skb_is_ulp_{ddp,crc} inline functions (Jakub).
- Fix copyright notice (Leon).
- Added missing ifdef to allow build with MLX5_EN_TLS disabled.
- Fix space alignment, indent and long lines (max 99 columns).
- Add missing field documentation in ulp_ddp.h.
Changes since v5:
=================
- Limit the series to RX offloads.
- Added two separated skb indications to avoid wrong flushing of GRO when aggerating offloaded packets.
- Use accessor functions for skb->ddp and skb->crc (Eric D) bits.
- Add kernel-doc for get_netdev_for_sock (Christoph).
- Remove ddp_iter* routines and only modify _copy_to_iter (Al Viro, Christoph).
- Remove consume skb (Sagi).
- Add a knob in the ddp limits struct for the HW driver to advertise if they need the nvme-tcp driver to apply the generation counter quirk. Use this knob for the mlx5 CX7 offload.
- bugfix: use u8 flags instead of bool in mlx5e_nvmeotcp_queue->dgst.
- bugfix: use sg_dma_len(sgl) instead of sgl->length.
- bugfix: remove sgl leak in nvme_tcp_setup_ddp().
- bugfix: remove sgl leak when only using DDGST_RX offload.
- Add error check for dma_map_sg().
- Reduce #ifdef by using dummy macros/functions.
- Remove redundant netdev null check in nvme_tcp_pdu_last_send().
- Rename ULP_DDP_RESYNC_{REQ -> PENDING}.
- Add per-ulp limits struct (Sagi).
- Add ULP DDP capabilities querying (Sagi).
- Simplify RX DDGST logic (Sagi).
- Document resync flow better.
- Add ulp_offload param to nvme-tcp module to enable ULP offload (Sagi).
- Add a revert commit to reintroduce nvme_tcp_queue->queue_size.
Changes since v4:
=================
- Add transmit offload patches.
- Use one feature bit for both receive and transmit offload.
Changes since v3:
=================
- Use DDP_TCP ifdefs in iov_iter and skb iterators to minimize impact when compiled out (Christoph).
- Simplify netdev references and reduce the use of get_netdev_for_sock (Sagi).
- Avoid "static" in it's own line, move it one line down (Christoph)
- Pass (queue, skb, *offset) and retrieve the pdu_seq in nvme_tcp_resync_response (Sagi).
- Add missing assignment of offloading_netdev to null in offload_limits error case (Sagi).
- Set req->offloaded = false once -- the lifetime rules are:
set to false on cmd_setup / set to true when ddp setup succeeds (Sagi).
- Replace pr_info_ratelimited with dev_info_ratelimited (Sagi).
- Add nvme_tcp_complete_request and invoke it from two similar call sites (Sagi).
- Introduce nvme_tcp_req_map_sg earlier in the series (Sagi).
- Add nvme_tcp_consume_skb and put into it a hunk from nvme_tcp_recv_data to handle copy with and without offload.
Changes since v2:
=================
- Use skb->ddp_crc for copy offload to avoid skb_condense.
- Default mellanox driver support to no (experimental feature).
- In iov_iter use non-ddp functions for kvec and iovec.
- Remove typecasting in NVMe-TCP.
Changes since v1:
=================
- Rework iov_iter copy skip if src==dst to be less intrusive (David Ahern).
- Add tcp-ddp documentation (David Ahern).
- Refactor mellanox driver patches into more patches (Saeed Mahameed).
- Avoid pointer casting (David Ahern).
- Rename NVMe-TCP offload flags (Shai Malin).
- Update cover-letter according to the above.
Changes since RFC v1:
=====================
- Split mlx5 driver patches to several commits.
- Fix NVMe-TCP handling of recovery flows. In particular, move queue offload. init/teardown to the start/stop functions.
Aurelien Aptel (7):
net/ethtool: add new stringset ETH_SS_ULP_DDP_{CAPS,STATS} net/ethtool: add ULP_DDP_{GET,SET} operations for caps and stats
Documentation: document netlink ULP_DDP_GET/SET messages net/tls,core: export get_netdev_for_sock
nvme-tcp: Add modparam to control the ULP offload enablement
nvme-tcp: Only enable offload with TLS if the driver supports it net/mlx5e: NVMEoTCP, statistics
Ben Ben-Ishay (8):
iov_iter: skip copy if src == dst for direct data placement net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations net/mlx5e: NVMEoTCP, offload initialization net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration net/mlx5e: NVMEoTCP, queue init/teardown
net/mlx5e: NVMEoTCP, ddp setup and resync
net/mlx5e: NVMEoTCP, async ddp invalidation net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload
Boris Pismenny (4):
net: Introduce direct data placement tcp offload
nvme-tcp: Add DDP offload control path
nvme-tcp: Add DDP data-path
net/mlx5e: TCP flow steering for nvme-tcp acceleration
Or Gerlitz (5):
nvme-tcp: Deal with netdevice DOWN events
net/mlx5e: Rename from tls to transport static params net/mlx5e: Refactor ico sq polling to get budget net/mlx5e: Have mdev pointer directly on the icosq structure net/mlx5e: Refactor doorbell function to allow avoiding a completion
Yoray Zack (2):
nvme-tcp: RX DDGST offload
Documentation: add ULP DDP offload documentation
Documentation/netlink/specs/ethtool.yaml | 102 ++ Documentation/networking/ethtool-netlink.rst | 92 ++ Documentation/networking/index.rst | 1 + Documentation/networking/statistics.rst | 1 + Documentation/networking/ulp-ddp-offload.rst | 378 ++++++ .../net/ethernet/mellanox/mlx5/core/Kconfig | 11 + .../net/ethernet/mellanox/mlx5/core/Makefile | 3 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 9 + .../net/ethernet/mellanox/mlx5/core/en/fs.h | 4 +- .../ethernet/mellanox/mlx5/core/en/params.c | 12 +- .../ethernet/mellanox/mlx5/core/en/params.h | 3 + .../mellanox/mlx5/core/en/reporter_rx.c | 4 +- .../ethernet/mellanox/mlx5/core/en/rx_res.c | 28 + .../ethernet/mellanox/mlx5/core/en/rx_res.h | 4 + .../net/ethernet/mellanox/mlx5/core/en/tir.c | 15 + .../net/ethernet/mellanox/mlx5/core/en/tir.h | 2 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 34 +- .../mlx5/core/en_accel/common_utils.h | 32 + .../mellanox/mlx5/core/en_accel/en_accel.h | 3 + .../mellanox/mlx5/core/en_accel/fs_tcp.c | 12 +- .../mellanox/mlx5/core/en_accel/fs_tcp.h | 2 +- .../mellanox/mlx5/core/en_accel/ktls.c | 2 +- .../mellanox/mlx5/core/en_accel/ktls_rx.c | 8 +- .../mellanox/mlx5/core/en_accel/ktls_tx.c | 8 +- .../mellanox/mlx5/core/en_accel/ktls_txrx.c | 36 +- .../mellanox/mlx5/core/en_accel/ktls_utils.h | 17 +- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 1117 +++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 142 +++ .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 354 ++++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.h | 37 + .../mlx5/core/en_accel/nvmeotcp_stats.c | 66 + .../mlx5/core/en_accel/nvmeotcp_utils.h | 66 + .../ethernet/mellanox/mlx5/core/en_ethtool.c | 6 + .../net/ethernet/mellanox/mlx5/core/en_fs.c | 4 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 30 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 71 +- .../ethernet/mellanox/mlx5/core/en_stats.h | 7 + .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 4 +- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + .../net/ethernet/mellanox/mlx5/core/main.c | 1 + drivers/nvme/host/tcp.c | 560 ++++++++- include/linux/ethtool.h | 32 + include/linux/mlx5/device.h | 59 +- include/linux/mlx5/mlx5_ifc.h | 82 +- include/linux/mlx5/qp.h | 1 + include/linux/netdevice.h | 18 +- include/linux/skbuff.h | 25 +- include/net/inet_connection_sock.h | 6 + include/net/ulp_ddp.h | 191 +++ include/net/ulp_ddp_caps.h | 35 + include/uapi/linux/ethtool.h | 4 + include/uapi/linux/ethtool_netlink.h | 40 + lib/iov_iter.c | 8 +- net/Kconfig | 20 + net/core/dev.c | 26 +- net/core/skbuff.c | 3 +- net/ethtool/Makefile | 2 +- net/ethtool/bitset.c | 20 +- net/ethtool/common.c | 23 + net/ethtool/common.h | 2 + net/ethtool/netlink.c | 20 + net/ethtool/netlink.h | 4 + net/ethtool/strset.c | 11 + net/ethtool/ulp_ddp.c | 316 +++++ net/ipv4/tcp_input.c | 13 +- net/ipv4/tcp_ipv4.c | 3 + net/ipv4/tcp_offload.c | 3 + net/tls/tls_device.c | 16 - 68 files changed, 4114 insertions(+), 163 deletions(-) create mode 100644 Documentation/networking/ulp-ddp-offload.rst create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h create mode 100644 include/net/ulp_ddp.h create mode 100644 include/net/ulp_ddp_caps.h create mode 100644 net/ethtool/ulp_ddp.c
Signature · 1 line
2.34.1
PATCH v12 1/26 net: Introduce direct data placement tcp offload
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement (DDP) offload for TCP.
The motivation is saving compute resources/cycles that are spent to copy data from SKBs to the block layer buffers and CRC calculation/verification for received PDUs (Protocol Data Units).
The DDP capability is accompanied by new net_device operations that configure hardware contexts.
There is a context per socket, and a context per DDP operation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments.
The interface includes the following net-device ddp operations:
1. sk_add - add offload for the queue represented by socket+config pair
2. sk_del - remove the offload for the socket/queue
3. ddp_setup - request copy offload for buffers associated with an IO
4. ddp_teardown - release offload resources for that IO
5. limits - query NIC driver for quirks and limitations (e.g. max number of scatter gather entries per IO)
6. set_caps - request ULP DDP capabilities enablement
7. get_stats - query NIC driver for ULP DDP stats
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.
The SKB passed to the network stack from the driver represents data as it is on the wire, while it is pointing directly to data in destination buffers.
As a result, data from page frags should not be copied out to the linear part. To avoid needless copies, such as when using skb_condense, we mark the skb->ulp_ddp bit.
In addition, the skb->ulp_crc will be used by the upper layers to determine if CRC re-calculation is required. The two separated skb indications are needed to avoid false positives GRO flushing events.
Follow-up patches will use this interface for DDP in NVMe-TCP.
Capability bits stored in net_device allow drivers to report which ULP DDP capabilities a device supports. Control over these capabilities will be exposed to userspace in later patches.
--- include/linux/netdevice.h | 15 +++ include/linux/skbuff.h | 25 +++- include/net/inet_connection_sock.h | 6 + include/net/ulp_ddp.h | 191 +++++++++++++++++++++++++++++ include/net/ulp_ddp_caps.h | 35 ++++++ net/Kconfig | 20 +++ net/core/skbuff.c | 3 +- net/ipv4/tcp_input.c | 13 +- net/ipv4/tcp_ipv4.c | 3 + net/ipv4/tcp_offload.c | 3 + 10 files changed, 311 insertions(+), 3 deletions(-) create mode 100644 include/net/ulp_ddp.h create mode 100644 include/net/ulp_ddp_caps.h
include/linux/netdevice.h +15 -0
@@ -54,6 +54,10 @@#include <net/net_debug.h> #include <net/dropreason-core.h> +#ifdef CONFIG_ULP_DDP +#include <net/ulp_ddp_caps.h> +#endif +struct netpoll_info ;struct device ;struct ethtool_ops ;
@@ -1418,6 +1422,8 @@ struct netdev_net_notifier {* Get hardware timestamp based on normal/adjustable time or free running * cycle counter. This function is required if physical clock supports a * free running cycle counter. +* struct ulp_ddp_dev_ops *ulp_ddp_ops; +* ULP DDP operations (see include/net/ulp_ddp.h) */ struct net_device_ops {int (*ndo_init)(struct net_device *dev);
@@ -1652,6 +1658,9 @@ struct net_device_ops {ktime_t (*ndo_get_tstamp)(struct net_device *dev,const struct skb_shared_hwtstamps *hwtstamps,bool cycles); +#if IS_ENABLED (CONFIG_ULP_DDP ) +const struct ulp_ddp_dev_ops *ulp_ddp_ops; +#endif };struct xdp_metadata_ops {
@@ -1825,6 +1834,9 @@ enum netdev_ml_priv_type {* @mpls_features: Mask of features inheritable by MPLS * @gso_partial_features: value(s) from NETIF_F_GSO\* * +* @ulp_ddp_caps: Bitflags keeping track of supported and enabled +* ULP DDP capabilities. +* * @ifindex: interface index * @group: The group the device belongs to *
@@ -2121,6 +2133,9 @@ struct net_device {netdev_features_t mpls_features;netdev_features_t gso_partial_features; +#ifdef CONFIG_ULP_DDP +struct ulp_ddp_netdev_caps ulp_ddp_caps; +#endif unsigned int min_mtu;unsigned int max_mtu;unsigned short type;
include/linux/skbuff.h +24 -1
@@ -813,6 +813,8 @@ typedef unsigned char *sk_buff_data_t;* delivery_time in mono clock base (i.e. EDT). Otherwise, the * skb->tstamp has the (rcv) timestamp at ingress and * delivery_time at egress. +* @ulp_ddp: DDP offloaded +* @ulp_crc: CRC offloaded * @napi_id: id of the NAPI struct this skb came from * @sender_cpu: (aka @napi_id) source CPU in XPS * @alloc_cpu: CPU which did the skb allocation.
@@ -992,7 +994,10 @@ struct sk_buff {#if IS_ENABLED (CONFIG_IP_SCTP ) __u8 csum_not_inet:1;#endif - +#ifdef CONFIG_ULP_DDP + __u8 ulp_ddp:1; + __u8 ulp_crc:1; +#endif #ifdef CONFIG_NET_SCHED __u16 tc_index;/* traffic control index */ #endif
@@ -5040,5 +5045,23 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb)ssize_t skb_splice_from_iter (struct sk_buff *skb,struct iov_iter *iter,ssize_t maxsize,gfp_t gfp); +static inline bool skb_is_ulp_ddp (struct sk_buff *skb) +{ +#ifdef CONFIG_ULP_DDP +return skb->ulp_ddp; +#else +return 0 ; +#endif +} + +static inline bool skb_is_ulp_crc (struct sk_buff *skb) +{ +#ifdef CONFIG_ULP_DDP +return skb->ulp_crc; +#else +return 0 ; +#endif +} +#endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */
include/net/inet_connection_sock.h +6 -0
@@ -68,6 +68,8 @@ struct inet_connection_sock_af_ops {* @icsk_ulp_ops Pluggable ULP control hook * @icsk_ulp_data ULP private data * @icsk_clean_acked Clean acked data hook +* @icsk_ulp_ddp_ops Pluggable ULP direct data placement control hook +* @icsk_ulp_ddp_data ULP direct data placement private data * @icsk_ca_state: Congestion control state * @icsk_retransmits: Number of unrecovered [RTO] timeouts * @icsk_pending: Scheduled timer event
@@ -98,6 +100,10 @@ struct inet_connection_sock {const struct tcp_ulp_ops *icsk_ulp_ops;void __rcu *icsk_ulp_data;void (*icsk_clean_acked)(struct sock *sk,u32 acked_seq); +#ifdef CONFIG_ULP_DDP +const struct ulp_ddp_ulp_ops *icsk_ulp_ddp_ops; +void __rcu *icsk_ulp_ddp_data; +#endif unsigned int (*icsk_sync_mss)(struct sock *sk,u32 pmtu); __u8 icsk_ca_state:5, icsk_ca_initialized:1,
include/net/ulp_ddp.h new file +191 -0
191 lines not on this page · raw message
include/net/ulp_ddp_caps.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* ulp_ddp.h +* Author: Aurelien Aptel <[email protected]> +* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +*/ +#ifndef _ULP_DDP_CAPS_H +#define _ULP_DDP_CAPS_H + +#include <linux/types.h> + +enum { +ULP_DDP_C_NVME_TCP_BIT , +ULP_DDP_C_NVME_TCP_DDGST_RX_BIT , + +/* add capabilities above */ +ULP_DDP_C_COUNT , +}; + +struct ulp_ddp_netdev_caps { + DECLARE_BITMAP(active, ULP_DDP_C_COUNT); + DECLARE_BITMAP(hw, ULP_DDP_C_COUNT); +}; + +static inline bool ulp_ddp_cap_turned_on (unsigned long *old,unsigned long *new,int bit_nr) +{ +return !test_bit (bit_nr, old) &&test_bit (bit_nr, new); +} + +static inline bool ulp_ddp_cap_turned_off (unsigned long *old,unsigned long *new,int bit_nr) +{ +return test_bit (bit_nr, old) && !test_bit (bit_nr, new); +} + +#endif
net/Kconfig +20 -0
@@ -503,4 +503,24 @@ config NETDEV_ADDR_LIST_TEST default KUNIT_ALL_TESTS depends on KUNIT +config ULP_DDP + bool "ULP direct data placement offload" + help + This feature provides a generic infrastructure for Direct + Data Placement (DDP) offload for Upper Layer Protocols (ULP, + such as NVMe-TCP). + + If the ULP and NIC driver supports it, the ULP code can + request the NIC to place ULP response data directly + into application memory, avoiding a costly copy. + + This infrastructure also allows for offloading the ULP data + integrity checks (e.g. data digest) that would otherwise + require another costly pass on the data we managed to avoid + copying. + + For more information, see + <file:Documentation/networking/ulp-ddp-offload.rst>. + + endif # if NET
net/core/skbuff.c +2 -1
@@ -75,6 +75,7 @@#include <net/mctp.h> #include <net/page_pool.h> #include <net/dropreason.h> +#include <net/ulp_ddp.h> #include <linux/uaccess.h> #include <trace/events/skb.h>
@@ -6505,7 +6506,7 @@ void skb_condense(struct sk_buff *skb) {if (skb->data_len) {if (skb->data_len > skb->end - skb->tail || -skb_cloned (skb)) +skb_cloned (skb) ||skb_is_ulp_ddp (skb))return ;/* Nice, we can free page frag(s) right now */
net/ipv4/tcp_input.c +12 -1
@@ -4677,7 +4677,10 @@ static bool tcp_try_coalesce(struct sock *sk,if (from->decrypted != to->decrypted)return false;#endif - +#ifdef CONFIG_ULP_DDP +if (skb_is_ulp_crc (from) !=skb_is_ulp_crc (to)) +return false; +#endif if (!skb_try_coalesce (to, from, fragstolen, &delta))return false;
@@ -5240,6 +5243,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,memcpy (nskb->cb, skb->cb,sizeof (skb->cb));#ifdef CONFIG_TLS_DEVICE nskb->decrypted = skb->decrypted; +#endif +#ifdef CONFIG_ULP_DDP + nskb->ulp_ddp = skb->ulp_ddp; + nskb->ulp_crc = skb->ulp_crc;#endif TCP_SKB_CB (nskb)->seq =TCP_SKB_CB (nskb)->end_seq = start;if (list)
@@ -5273,6 +5280,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,#ifdef CONFIG_TLS_DEVICE if (skb->decrypted != nskb->decrypted) goto end; +#endif +#ifdef CONFIG_ULP_DDP +if (skb_is_ulp_crc (skb) !=skb_is_ulp_crc (nskb)) + goto end;#endif } }
net/ipv4/tcp_ipv4.c +3 -0
@@ -1866,6 +1866,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb, TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_ECE | TCPHDR_CWR)) ||#ifdef CONFIG_TLS_DEVICE tail->decrypted != skb->decrypted || +#endif +#ifdef CONFIG_ULP_DDP +skb_is_ulp_crc (tail) !=skb_is_ulp_crc (skb) || #endif thtail->doff != th->doff || memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)))
net/ipv4/tcp_offload.c +3 -0
@@ -268,6 +268,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)#ifdef CONFIG_TLS_DEVICE flush |= p->decrypted ^ skb->decrypted;#endif +#ifdef CONFIG_ULP_DDP + flush |=skb_is_ulp_crc (p) ^skb_is_ulp_crc (skb); +#endif if (flush ||skb_gro_receive (p, skb)) { mss =1 ;
Signature · 1 line
2.34.1
Sagi Grimberg
maintainerRe: [PATCH v12 01/26] net: Introduce direct data placement tcp offload
Quoted 239 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement (DDP) offload for TCP.
The motivation is saving compute resources/cycles that are spent to copy data from SKBs to the block layer buffers and CRC calculation/verification for received PDUs (Protocol Data Units).
The DDP capability is accompanied by new
net_deviceoperations that configure hardware contexts.There is a context per socket, and a context per DDP operation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments.
The interface includes the following net-device ddp operations:
1.
sk_add- add offload for the queue represented by socket+config pair
2.sk_del- remove the offload for the socket/queue
3.ddp_setup- request copy offload for buffers associated with an IO
4.ddp_teardown- release offload resources for that IO
5. limits - query NIC driver for quirks and limitations (e.g. max number of scatter gather entries per IO)
6.set_caps- request ULP DDP capabilities enablement
7.get_stats- query NIC driver for ULP DDP statsUsing this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the
command_id.To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.
The SKB passed to the network stack from the driver represents data as it is on the wire, while it is pointing directly to data in destination buffers.
As a result, data from page frags should not be copied out to the linear part. To avoid needless copies, such as when using
skb_condense, we mark theskb->ulp_ddpbit.
In addition, theskb->ulp_crcwill be used by the upper layers to determine if CRC re-calculation is required. The two separated skb indications are needed to avoid false positives GRO flushing events.Follow-up patches will use this interface for DDP in NVMe-TCP.
Capability bits stored in
net_deviceallow drivers to report which ULP DDP capabilities a device supports. Control over these capabilities will be exposed to userspace in later patches.Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>--- include/linux/netdevice.h | 15 +++ include/linux/skbuff.h | 25 +++- include/net/inet_connection_sock.h | 6 + include/net/ulp_ddp.h | 191 +++++++++++++++++++++++++++++ include/net/ulp_ddp_caps.h | 35 ++++++ net/Kconfig | 20 +++ net/core/skbuff.c | 3 +- net/ipv4/tcp_input.c | 13 +- net/ipv4/tcp_ipv4.c | 3 + net/ipv4/tcp_offload.c | 3 + 10 files changed, 311 insertions(+), 3 deletions(-)create mode 100644
include/net/ulp_ddp.h
create mode 100644include/net/ulp_ddp_caps.hinclude/linux/netdevice.h +15
-0@@ -54,6 +54,10 @@#include <net/net_debug.h> #include <net/dropreason-core.h> +#ifdef CONFIG_ULP_DDP +#include <net/ulp_ddp_caps.h> +#endif +struct netpoll_info ;struct device ;struct ethtool_ops ;@@ -1418,6 +1422,8 @@ struct netdev_net_notifier {* Get hardware timestamp based on normal/adjustable time or free running * cycle counter. This function is required if physical clock supports a * free running cycle counter. +* struct ulp_ddp_dev_ops *ulp_ddp_ops; +* ULP DDP operations (see include/net/ulp_ddp.h) */ struct net_device_ops {int (*ndo_init)(struct net_device *dev);@@ -1652,6 +1658,9 @@ struct net_device_ops {ktime_t (*ndo_get_tstamp)(struct net_device *dev,const struct skb_shared_hwtstamps *hwtstamps,bool cycles); +#if IS_ENABLED (CONFIG_ULP_DDP ) +const struct ulp_ddp_dev_ops *ulp_ddp_ops; +#endif };struct xdp_metadata_ops {@@ -1825,6 +1834,9 @@ enum netdev_ml_priv_type {* @mpls_features: Mask of features inheritable by MPLS * @gso_partial_features: value(s) from NETIF_F_GSO\* * +* @ulp_ddp_caps: Bitflags keeping track of supported and enabled +* ULP DDP capabilities. +* * @ifindex: interface index * @group: The group the device belongs to * @@ -2121,6 +2133,9 @@ struct net_device {netdev_features_t mpls_features;netdev_features_t gso_partial_features; +#ifdef CONFIG_ULP_DDP +struct ulp_ddp_netdev_caps ulp_ddp_caps; +#endif unsigned int min_mtu;unsigned int max_mtu;unsigned short type;include/linux/skbuff.h +24
-1@@ -813,6 +813,8 @@ typedef unsigned char *sk_buff_data_t;* delivery_time in mono clock base (i.e. EDT). Otherwise, the * skb->tstamp has the (rcv) timestamp at ingress and * delivery_time at egress. +* @ulp_ddp: DDP offloaded +* @ulp_crc: CRC offloaded * @napi_id: id of the NAPI struct this skb came from * @sender_cpu: (aka @napi_id) source CPU in XPS * @alloc_cpu: CPU which did the skb allocation. @@ -992,7 +994,10 @@ struct sk_buff {#if IS_ENABLED (CONFIG_IP_SCTP ) __u8 csum_not_inet:1;#endif - +#ifdef CONFIG_ULP_DDP + __u8 ulp_ddp:1; + __u8 ulp_crc:1; +#endif #ifdef CONFIG_NET_SCHED __u16 tc_index;/* traffic control index */ #endif @@ -5040,5 +5045,23 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb)ssize_t skb_splice_from_iter (struct sk_buff *skb,struct iov_iter *iter,ssize_t maxsize,gfp_t gfp); +static inline bool skb_is_ulp_ddp (struct sk_buff *skb) +{ +#ifdef CONFIG_ULP_DDP +return skb->ulp_ddp; +#else +return 0 ; +#endif +} + +static inline bool skb_is_ulp_crc (struct sk_buff *skb) +{ +#ifdef CONFIG_ULP_DDP +return skb->ulp_crc; +#else +return 0 ; +#endif +} +#endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ include/net/inet_connection_sock.h +6
-0@@ -68,6 +68,8 @@ struct inet_connection_sock_af_ops {* @icsk_ulp_ops Pluggable ULP control hook * @icsk_ulp_data ULP private data * @icsk_clean_acked Clean acked data hook +* @icsk_ulp_ddp_ops Pluggable ULP direct data placement control hook +* @icsk_ulp_ddp_data ULP direct data placement private data * @icsk_ca_state: Congestion control state * @icsk_retransmits: Number of unrecovered [RTO] timeouts * @icsk_pending: Scheduled timer event @@ -98,6 +100,10 @@ struct inet_connection_sock {const struct tcp_ulp_ops *icsk_ulp_ops;void __rcu *icsk_ulp_data;void (*icsk_clean_acked)(struct sock *sk,u32 acked_seq); +#ifdef CONFIG_ULP_DDP +const struct ulp_ddp_ulp_ops *icsk_ulp_ddp_ops; +void __rcu *icsk_ulp_ddp_data; +#endif unsigned int (*icsk_sync_mss)(struct sock *sk,u32 pmtu); __u8 icsk_ca_state:5, icsk_ca_initialized:1,include/net/ulp_ddp.h new file +34
-0@@ -0,0 +1,191 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* ulp_ddp.h +* Author: Boris Pismenny <[email protected]> +* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +*/ +#ifndef _ULP_DDP_H +#define _ULP_DDP_H + +#include <linux/netdevice.h> +#include <net/inet_connection_sock.h> +#include <net/sock.h> + +#include "ulp_ddp_caps.h" + +enum ulp_ddp_type { +ULP_DDP_NVME =1 , +}; + +/** +* struct nvme_tcp_ddp_limits - nvme tcp driver limitations +* +* @full_ccid_range: true if the driver supports the full CID range +*/ +struct nvme_tcp_ddp_limits { +bool full_ccid_range; +}; + +/** +* struct ulp_ddp_limits - Generic ulp ddp limits: tcp ddp +* protocol limits. +* Add new instances of ulp_ddp_limits in the union below (nvme-tcp, etc.). +* +* @type: type of this limits struct
include/net/ulp_ddp.h line 44
Quoted 10 lines
include/net/ulp_ddp.h new file +10
-0+* @max_ddp_sgl_len: maximum sgl size supported (zero means no limit) +* @io_threshold: minimum payload size required to offload +* @tls: support for ULP over TLS +* @nvmeotcp: NVMe-TCP specific limits +*/ +struct ulp_ddp_limits { +enum ulp_ddp_type type; +int max_ddp_sgl_len; +int io_threshold; +bool tls:1 ;
Is this a catch-all for tls 1.2/1.3 or future ones?
Quoted 89 lines
include/net/ulp_ddp.h new file +89
-0+union { +struct nvme_tcp_ddp_limits nvmeotcp; + }; +}; + +/** +* struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue +* +* @pfv: pdu version (e.g., NVME_TCP_PFV_1_0) +* @cpda: controller pdu data alignment (dwords, 0's based) +* @dgst: digest types enabled (header or data, see enum nvme_tcp_digest_option). +* The netdev will offload crc if it is supported. +* @queue_size: number of nvme-tcp IO queue elements +* @queue_id: queue identifier +* @io_cpu: cpu core running the IO thread for this queue +*/ +struct nvme_tcp_ddp_config { +u16 pfv; +u8 cpda; +u8 dgst; +int queue_size; +int queue_id; +int io_cpu; +}; + +/** +* struct ulp_ddp_config - Generic ulp ddp configuration +* Add new instances of ulp_ddp_config in the union below (nvme-tcp, etc.). +* +* @type: type of this config struct +* @nvmeotcp: NVMe-TCP specific config +*/ +struct ulp_ddp_config { +enum ulp_ddp_type type; +union { +struct nvme_tcp_ddp_config nvmeotcp; + }; +}; + +/** +* struct ulp_ddp_io - ulp ddp configuration for an IO request. +* +* @command_id: identifier on the wire associated with these buffers +* @nents: number of entries in the sg_table +* @sg_table: describing the buffers for this IO request +* @first_sgl: first SGL in sg_table +*/ +struct ulp_ddp_io { +u32 command_id; +int nents; +struct sg_table sg_table; +struct scatterlist first_sgl[SG_CHUNK_SIZE ]; +}; + +struct ethtool_ulp_ddp_stats ; +struct netlink_ext_ack ; + +/** +* struct ulp_ddp_dev_ops - operations used by an upper layer protocol +* to configure ddp offload +* +* @limits: query ulp driver limitations and quirks. +* @sk_add: add offload for the queue represented by socket+config +* pair. this function is used to configure either copy, crc +* or both offloads. +* @sk_del: remove offload from the socket, and release any device +* related resources. +* @setup: request copy offload for buffers associated with a +* command_id in ulp_ddp_io. +* @teardown: release offload resources association between buffers +* and command_id in ulp_ddp_io. +* @resync: respond to the driver's resync_request. Called only if +* resync is successful. +* @set_caps: set device ULP DDP capabilities. +* returns a negative error code or zero. +* @get_stats: query ULP DDP statistics. +*/ +struct ulp_ddp_dev_ops { +int (*limits)(struct net_device *netdev, +struct ulp_ddp_limits *limits); +int (*sk_add)(struct net_device *netdev, +struct sock *sk, +struct ulp_ddp_config *config); +void (*sk_del)(struct net_device *netdev, +struct sock *sk); +int (*setup)(struct net_device *netdev, +struct sock *sk, +struct ulp_ddp_io *io); + void (*teardown)(struct net_device *netdev,
include/net/ulp_ddp.h line 143
Quoted 10 lines
include/net/ulp_ddp.h new file +10
-0+struct sock *sk, + struct ulp_ddp_io *io, + void *ddp_ctx); +void (*resync)(struct net_device *netdev, +struct sock *sk,u32 seq); +int (*set_caps)(struct net_device *dev,unsigned long *bits, +struct netlink_ext_ack *extack); +int (*get_stats)(struct net_device *dev, +struct ethtool_ulp_ddp_stats *stats); +};
It would be beneficial to have proper wrappers to these that can also do some housekeeping work around the callbacks.
Re: [PATCH v12 01/26] net: Introduce direct data placement tcp offload
Quoted 7 lines · Sagi Grimberg
include/net/ulp_ddp.h line 44
Quoted 5 lines
include/net/ulp_ddp.h new file +5
-0+struct ulp_ddp_limits { +enum ulp_ddp_type type; +int max_ddp_sgl_len; +int io_threshold; +bool tls:1 ;Is this a catch-all for tls 1.2/1.3 or future ones?
This is catch-all. Once it will be supported together with the offload, we can add the TLS type incrementaly.
Quoted 17 lines
Quoted 17 lines
include/net/ulp_ddp.h new file +17
-0+struct ulp_ddp_dev_ops { +int (*limits)(struct net_device *netdev, +struct ulp_ddp_limits *limits); +int (*sk_add)(struct net_device *netdev, +struct sock *sk, +struct ulp_ddp_config *config); +void (*sk_del)(struct net_device *netdev, +struct sock *sk); +int (*setup)(struct net_device *netdev, +struct sock *sk, +struct ulp_ddp_io *io); +void (*teardown)(struct net_device *netdev, +struct sock *sk, +struct ulp_ddp_io *io, +void *ddp_ctx); +void (*resync)(struct net_device *netdev, +struct sock *sk,u32 seq);
Quoted 8 lines
include/net/ulp_ddp.h line 143
Quoted 5 lines
include/net/ulp_ddp.h new file +5
-0+int (*set_caps)(struct net_device *dev,unsigned long *bits, +struct netlink_ext_ack *extack); +int (*get_stats)(struct net_device *dev, +struct ethtool_ulp_ddp_stats *stats); +};It would be beneficial to have proper wrappers to these that can also do some housekeeping work around the callbacks.
We can add wrappers in net/core/ulp_ddp.c that can NULL-check the function pointers and do any common housekeeping if needed.
Thanks
PATCH v12 2/26 net/ethtool: add new stringset ETH_SS_ULP_DDP_{CAPS,STATS}
In reply to Aurelien AptelThis commit exposes ULP DDP capability and statistics names to userspace via netlink.
In order to support future ULP DDP capabilities and statistics without having to change the netlink protocol (and userspace ethtool) we add new string sets to let userspace dynamically fetch what the kernel supports.
* ETH_SS_ULP_DDP_CAPS stores names of ULP DDP capabilities
* ETH_SS_ULP_DDP_STATS stores names of ULP DDP statistics.
These stringsets will be used in later commits when implementing the new ULP DDP GET/SET netlink messages.
We keep the convention of strset.c of having the static_assert() right after the array declaration, despite the checkpatch warning.
--- include/linux/ethtool.h | 32 ++++++++++++++++++++++++++++ include/uapi/linux/ethtool.h | 4 ++++ include/uapi/linux/ethtool_netlink.h | 22 +++++++++++++++++++ net/ethtool/common.c | 23 ++++++++++++++++++++ net/ethtool/common.h | 2 ++ net/ethtool/strset.c | 11 ++++++++++ 6 files changed, 94 insertions(+)
include/linux/ethtool.h +32 -0
@@ -594,6 +594,38 @@ struct ethtool_mm_stats {u64 MACMergeHoldCount; }; +/** +* struct ethtool_ulp_ddp_stats - ULP DDP offload statistics +* @rx_nvmeotcp_sk_add: number of sockets successfully prepared for offloading. +* @rx_nvmeotcp_sk_add_fail: number of sockets that failed to be prepared for offloading. +* @rx_nvmeotcp_sk_del: number of sockets where offloading has been removed. +* @rx_nvmeotcp_ddp_setup: number of NVMeTCP PDU successfully prepared for Direct Data Placement. +* @rx_nvmeotcp_ddp_setup_fail: number of PDUs that failed DDP preparation. +* @rx_nvmeotcp_ddp_teardown: number of PDUs done with DDP. +* @rx_nvmeotcp_drop: number of PDUs dropped. +* @rx_nvmeotcp_resync: number of resync. +* @rx_nvmeotcp_packets: number of offloaded PDUs. +* @rx_nvmeotcp_bytes: number of offloaded bytes. +*/ +struct ethtool_ulp_ddp_stats { +u64 rx_nvmeotcp_sk_add; +u64 rx_nvmeotcp_sk_add_fail; +u64 rx_nvmeotcp_sk_del; +u64 rx_nvmeotcp_ddp_setup; +u64 rx_nvmeotcp_ddp_setup_fail; +u64 rx_nvmeotcp_ddp_teardown; +u64 rx_nvmeotcp_drop; +u64 rx_nvmeotcp_resync; +u64 rx_nvmeotcp_packets; +u64 rx_nvmeotcp_bytes; + +/* +* add new stats at the end and keep in sync with +* - ETHTOOL_ULP_DDP_STATS_* enum in uapi +* - ulp_ddp_stats_name stringset +*/ +}; +/** * struct ethtool_ops - optional netdev operations * @cap_link_lanes_supported: indicates if the driver supports lanes
include/uapi/linux/ethtool.h +4 -0
@@ -681,6 +681,8 @@ enum ethtool_link_ext_substate_module {* @ETH_SS_STATS_ETH_MAC: names of IEEE 802.3 MAC statistics * @ETH_SS_STATS_ETH_CTRL: names of IEEE 802.3 MAC Control statistics * @ETH_SS_STATS_RMON: names of RMON statistics +* @ETH_SS_ULP_DDP_CAPS: names of ULP DDP capabilities +* @ETH_SS_ULP_DDP_STATS: names of ULP DDP statistics * * @ETH_SS_COUNT: number of defined string sets */
@@ -706,6 +708,8 @@ enum ethtool_stringset {ETH_SS_STATS_ETH_MAC ,ETH_SS_STATS_ETH_CTRL ,ETH_SS_STATS_RMON , +ETH_SS_ULP_DDP_CAPS , +ETH_SS_ULP_DDP_STATS ,/* add new constants above here */ ETH_SS_COUNT
include/uapi/linux/ethtool_netlink.h +22 -0
@@ -975,6 +975,28 @@ enum { ETHTOOL_A_MM_MAX = (__ETHTOOL_A_MM_CNT - 1) }; +/* ULP DDP */ + +enum { +ETHTOOL_A_ULP_DDP_STATS_UNSPEC , +ETHTOOL_A_ULP_DDP_STATS_PAD , + +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD_FAIL , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_DEL , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP_FAIL , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_TEARDOWN , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DROP , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_RESYNC , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_PACKETS , +ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_BYTES , + +/* add new constants above here */ + __ETHTOOL_A_ULP_DDP_STATS_CNT, +ETHTOOL_A_ULP_DDP_STATS_MAX = __ETHTOOL_A_ULP_DDP_STATS_CNT -1 +}; +/* generic netlink info */ #define ETHTOOL_GENL_NAME "ethtool"#define ETHTOOL_GENL_VERSION 1
net/ethtool/common.c +23 -0
@@ -5,6 +5,7 @@#include <linux/phy.h> #include <linux/rtnetlink.h> #include <linux/ptp_clock_kernel.h> +#include <net/ulp_ddp_caps.h> #include "common.h"
@@ -465,6 +466,28 @@ const char udp_tunnel_type_names[][ETH_GSTRING_LEN] = {static_assert (ARRAY_SIZE (udp_tunnel_type_names) == __ETHTOOL_UDP_TUNNEL_TYPE_CNT); +const char ulp_ddp_caps_names[][ETH_GSTRING_LEN ] = { + [ULP_DDP_C_NVME_TCP_BIT ] ="nvme-tcp-ddp" , + [ULP_DDP_C_NVME_TCP_DDGST_RX_BIT ] ="nvme-tcp-ddgst-rx-offload" , +}; +static_assert (ARRAY_SIZE (ulp_ddp_caps_names) ==ULP_DDP_C_COUNT ); + +const char ulp_ddp_stats_names[][ETH_GSTRING_LEN ] = { + [ETHTOOL_A_ULP_DDP_STATS_UNSPEC ] ="unspec" , + [ETHTOOL_A_ULP_DDP_STATS_PAD ] ="pad" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD ] ="rx_nvmeotcp_sk_add" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD_FAIL ] ="rx_nvmeotcp_sk_add_fail" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_DEL ] ="rx_nvmeotcp_sk_del" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP ] ="rx_nvmeotcp_ddp_setup" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP_FAIL ] ="rx_nvmeotcp_ddp_setup_fail" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_TEARDOWN ] ="rx_nvmeotcp_ddp_teardown" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DROP ] ="rx_nvmeotcp_drop" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_RESYNC ] ="rx_nvmeotcp_resync" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_PACKETS ] ="rx_nvmeotcp_packets" , + [ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_BYTES ] ="rx_nvmeotcp_bytes" , +}; +static_assert (ARRAY_SIZE (ulp_ddp_stats_names) == __ETHTOOL_A_ULP_DDP_STATS_CNT); +/* return false if legacy contained non-0 deprecated fields * maxtxpkt/maxrxpkt. rest of ksettings always updated */
net/ethtool/common.h +2 -0
@@ -36,6 +36,8 @@ extern const char sof_timestamping_names[][ETH_GSTRING_LEN];extern const char ts_tx_type_names[][ETH_GSTRING_LEN ];extern const char ts_rx_filter_names[][ETH_GSTRING_LEN ];extern const char udp_tunnel_type_names[][ETH_GSTRING_LEN ]; +extern const char ulp_ddp_caps_names[][ETH_GSTRING_LEN ]; +extern const char ulp_ddp_stats_names[][ETH_GSTRING_LEN ];int __ethtool_get_link (struct net_device *dev);
net/ethtool/strset.c +11 -0
@@ -2,6 +2,7 @@#include <linux/ethtool.h> #include <linux/phy.h> +#include <net/ulp_ddp_caps.h> #include "netlink.h" #include "common.h"
@@ -105,6 +106,16 @@ static const struct strset_info info_template[] = { .count = __ETHTOOL_A_STATS_RMON_CNT, .strings = stats_rmon_names, }, + [ETH_SS_ULP_DDP_CAPS] = { + .per_dev = false, + .count = ULP_DDP_C_COUNT, + .strings = ulp_ddp_caps_names, + }, + [ETH_SS_ULP_DDP_STATS] = { + .per_dev = false, + .count = __ETHTOOL_A_ULP_DDP_STATS_CNT, + .strings = ulp_ddp_stats_names, + }, };struct strset_req_info {
Signature · 1 line
2.34.1
PATCH v12 3/26 net/ethtool: add ULP_DDP_{GET,SET} operations for caps and stats
In reply to Aurelien AptelThis commit adds:
- 3 new netlink messages:
* ULP_DDP_GET: returns a bitset of supported and active capabilities
* ULP_DDP_SET: tries to activate requested bitset and returns results
* ULP_DDP_NTF: notification for capabilities change
Rename and export bitset_policy for use in ulp_ddp.c.
ULP DDP capabilities handling is similar to netdev features handling.
If a ULP_DDP_GET message has requested statistics via the ETHTOOL_FLAG_STATS header flag, then statistics are returned to userspace.
ULP_DDP_GET request: (header only)
ULP_DDP_GET reply:
HW (bitset)
ACTIVE (bitset)
STATS (nest, optional)
STATS_xxxx (u64)
.... ULP_DDP_SET request:
WANTED (bitset)
ULP_DDP_SET reply:
WANTED (bitset)
ACTIVE (bitset)
--- include/uapi/linux/ethtool_netlink.h | 18 ++ net/ethtool/Makefile | 2 +- net/ethtool/bitset.c | 20 +- net/ethtool/netlink.c | 20 ++ net/ethtool/netlink.h | 4 + net/ethtool/ulp_ddp.c | 316 +++++++++++++++++++++++++++ 6 files changed, 369 insertions(+), 11 deletions(-) create mode 100644 net/ethtool/ulp_ddp.c
include/uapi/linux/ethtool_netlink.h +18 -0
@@ -57,6 +57,8 @@ enum {ETHTOOL_MSG_PLCA_GET_STATUS ,ETHTOOL_MSG_MM_GET ,ETHTOOL_MSG_MM_SET , +ETHTOOL_MSG_ULP_DDP_GET , +ETHTOOL_MSG_ULP_DDP_SET ,/* add new constants above here */ __ETHTOOL_MSG_USER_CNT,
@@ -109,6 +111,9 @@ enum {ETHTOOL_MSG_PLCA_NTF ,ETHTOOL_MSG_MM_GET_REPLY ,ETHTOOL_MSG_MM_NTF , +ETHTOOL_MSG_ULP_DDP_GET_REPLY , +ETHTOOL_MSG_ULP_DDP_SET_REPLY , +ETHTOOL_MSG_ULP_DDP_NTF ,/* add new constants above here */ __ETHTOOL_MSG_KERNEL_CNT,
@@ -977,6 +982,19 @@ enum {/* ULP DDP */ +enum { +ETHTOOL_A_ULP_DDP_UNSPEC , +ETHTOOL_A_ULP_DDP_HEADER ,/* nest - _A_HEADER_* */ +ETHTOOL_A_ULP_DDP_HW ,/* bitset */ +ETHTOOL_A_ULP_DDP_ACTIVE ,/* bitset */ +ETHTOOL_A_ULP_DDP_WANTED ,/* bitset */ +ETHTOOL_A_ULP_DDP_STATS ,/* nest - _A_ULP_DDP_STATS_* */ + +/* add new constants above here */ + __ETHTOOL_A_ULP_DDP_CNT, +ETHTOOL_A_ULP_DDP_MAX = __ETHTOOL_A_ULP_DDP_CNT -1 +}; +enum {ETHTOOL_A_ULP_DDP_STATS_UNSPEC ,ETHTOOL_A_ULP_DDP_STATS_PAD ,
net/ethtool/Makefile +1 -1
@@ -8,4 +8,4 @@ ethtool_nl-y := netlink.o bitset.o strset.o linkinfo.o linkmodes.o rss.o \ linkstate.o debug.o wol.o features.o privflags.o rings.o \ channels.o coalesce.o pause.o eee.o tsinfo.o cabletest.o \ tunnels.o fec.o eeprom.o stats.o phc_vclocks.o mm.o \ - module.o pse-pd.o plca.o mm.o + module.o pse-pd.o plca.o ulp_ddp.o
net/ethtool/bitset.c +10 -10
@@ -302,7 +302,7 @@ int ethnl_put_bitset32(struct sk_buff *skb, int attrtype, const u32 *val,return -EMSGSIZE ; } -static const struct nla_policy bitset_policy[] = { +const struct nla_policy ethnl_bitset_policy[] = { [ETHTOOL_A_BITSET_NOMASK ] = { .type =NLA_FLAG }, [ETHTOOL_A_BITSET_SIZE ] =NLA_POLICY_MAX (NLA_U32 ,ETHNL_MAX_BITSET_SIZE ),
@@ -327,11 +327,11 @@ static const struct nla_policy bit_policy[] = {*/ int ethnl_bitset_is_compact (const struct nlattr *bitset,bool *compact) { -struct nlattr *tb[ARRAY_SIZE (bitset_policy)]; +struct nlattr *tb[ARRAY_SIZE (ethnl_bitset_policy)];int ret; - ret =nla_parse_nested (tb,ARRAY_SIZE (bitset_policy) -1 , bitset, - bitset_policy,NULL ); + ret =nla_parse_nested (tb,ARRAY_SIZE (ethnl_bitset_policy) -1 , bitset, + ethnl_bitset_policy,NULL );if (ret <0 )return ret;
@@ -553,15 +553,15 @@ int ethnl_update_bitset32(u32 *bitmap, unsigned int nbits, const struct nlattr *attr, ethnl_string_array_t names, struct netlink_ext_ack *extack, bool *mod) { - struct nlattr *tb[ARRAY_SIZE(bitset_policy)]; + struct nlattr *tb[ARRAY_SIZE(ethnl_bitset_policy)];unsigned int change_bits;bool no_mask;int ret;if (!attr)return 0 ; - ret =nla_parse_nested (tb,ARRAY_SIZE (bitset_policy) -1 , attr, - bitset_policy, extack); + ret =nla_parse_nested (tb,ARRAY_SIZE (ethnl_bitset_policy) -1 , attr, + ethnl_bitset_policy, extack);if (ret <0 )return ret;
@@ -606,7 +606,7 @@ int ethnl_parse_bitset(unsigned long *val, unsigned long *mask,ethnl_string_array_t names, struct netlink_ext_ack *extack) { - struct nlattr *tb[ARRAY_SIZE(bitset_policy)]; + struct nlattr *tb[ARRAY_SIZE(ethnl_bitset_policy)];const struct nlattr *bit_attr;bool no_mask;int rem;
@@ -614,8 +614,8 @@ int ethnl_parse_bitset(unsigned long *val, unsigned long *mask,if (!attr)return 0 ; - ret =nla_parse_nested (tb,ARRAY_SIZE (bitset_policy) -1 , attr, - bitset_policy, extack); + ret =nla_parse_nested (tb,ARRAY_SIZE (ethnl_bitset_policy) -1 , attr, + ethnl_bitset_policy, extack);if (ret <0 )return ret; no_mask = tb[ETHTOOL_A_BITSET_NOMASK ];
net/ethtool/netlink.c +20 -0
@@ -308,6 +308,8 @@ ethnl_default_requests[__ETHTOOL_MSG_USER_CNT] = { [ETHTOOL_MSG_PLCA_GET_STATUS] = ðnl_plca_status_request_ops, [ETHTOOL_MSG_MM_GET] = ðnl_mm_request_ops, [ETHTOOL_MSG_MM_SET] = ðnl_mm_request_ops, + [ETHTOOL_MSG_ULP_DDP_GET] = ðnl_ulp_ddp_request_ops, + [ETHTOOL_MSG_ULP_DDP_SET] = ðnl_ulp_ddp_request_ops, };static struct ethnl_dump_ctx *ethnl_dump_context (struct netlink_callback *cb)
@@ -671,6 +673,7 @@ ethnl_default_notify_ops[ETHTOOL_MSG_KERNEL_MAX + 1] = { [ETHTOOL_MSG_MODULE_NTF] = ðnl_module_request_ops, [ETHTOOL_MSG_PLCA_NTF] = ðnl_plca_cfg_request_ops, [ETHTOOL_MSG_MM_NTF] = ðnl_mm_request_ops, + [ETHTOOL_MSG_ULP_DDP_NTF] = ðnl_ulp_ddp_request_ops, };/* default notification handler */
@@ -766,6 +769,7 @@ static const ethnl_notify_handler_t ethnl_notify_handlers[] = { [ETHTOOL_MSG_MODULE_NTF] = ethnl_default_notify, [ETHTOOL_MSG_PLCA_NTF] = ethnl_default_notify, [ETHTOOL_MSG_MM_NTF] = ethnl_default_notify, + [ETHTOOL_MSG_ULP_DDP_NTF] = ethnl_default_notify, };void ethtool_notify (struct net_device *dev,unsigned int cmd,const void *data)
@@ -1158,6 +1162,22 @@ static const struct genl_ops ethtool_genl_ops[] = { .policy = ethnl_mm_set_policy, .maxattr = ARRAY_SIZE(ethnl_mm_set_policy) - 1, }, + { + .cmd = ETHTOOL_MSG_ULP_DDP_GET, + .doit = ethnl_default_doit, + .start = ethnl_default_start, + .dumpit = ethnl_default_dumpit, + .done = ethnl_default_done, + .policy = ethnl_ulp_ddp_get_policy, + .maxattr = ARRAY_SIZE(ethnl_ulp_ddp_get_policy) - 1, + }, + { + .cmd = ETHTOOL_MSG_ULP_DDP_SET, + .flags = GENL_UNS_ADMIN_PERM, + .doit = ethnl_default_set_doit, + .policy = ethnl_ulp_ddp_set_policy, + .maxattr = ARRAY_SIZE(ethnl_ulp_ddp_set_policy) - 1, + }, }; static const struct genl_multicast_group ethtool_nl_mcgrps[] = {
net/ethtool/netlink.h +4 -0
@@ -395,10 +395,12 @@ extern const struct ethnl_request_ops ethnl_rss_request_ops;extern const struct ethnl_request_ops ethnl_plca_cfg_request_ops;extern const struct ethnl_request_ops ethnl_plca_status_request_ops;extern const struct ethnl_request_ops ethnl_mm_request_ops; +extern const struct ethnl_request_ops ethnl_ulp_ddp_request_ops;extern const struct nla_policy ethnl_header_policy[ETHTOOL_A_HEADER_FLAGS +1 ];extern const struct nla_policy ethnl_header_policy_stats[ETHTOOL_A_HEADER_FLAGS +1 ];extern const struct nla_policy ethnl_strset_get_policy[ETHTOOL_A_STRSET_COUNTS_ONLY +1 ]; +extern const struct nla_policy ethnl_bitset_policy[ETHTOOL_A_BITSET_MASK +1 ];extern const struct nla_policy ethnl_linkinfo_get_policy[ETHTOOL_A_LINKINFO_HEADER +1 ];extern const struct nla_policy ethnl_linkinfo_set_policy[ETHTOOL_A_LINKINFO_TP_MDIX_CTRL +1 ];extern const struct nla_policy ethnl_linkmodes_get_policy[ETHTOOL_A_LINKMODES_HEADER +1 ];
@@ -441,6 +443,8 @@ extern const struct nla_policy ethnl_plca_set_cfg_policy[ETHTOOL_A_PLCA_MAX + 1]extern const struct nla_policy ethnl_plca_get_status_policy[ETHTOOL_A_PLCA_HEADER +1 ];extern const struct nla_policy ethnl_mm_get_policy[ETHTOOL_A_MM_HEADER +1 ];extern const struct nla_policy ethnl_mm_set_policy[ETHTOOL_A_MM_MAX +1 ]; +extern const struct nla_policy ethnl_ulp_ddp_get_policy[ETHTOOL_A_ULP_DDP_HEADER +1 ]; +extern const struct nla_policy ethnl_ulp_ddp_set_policy[ETHTOOL_A_ULP_DDP_WANTED +1 ];int ethnl_set_features (struct sk_buff *skb,struct genl_info *info);int ethnl_act_cable_test (struct sk_buff *skb,struct genl_info *info);
net/ethtool/ulp_ddp.c new file +316 -0
316 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v12 03/26] net/ethtool: add ULP_DDP_{GET,SET} operations for caps and stats
On Wed, Jul 12, 2023 at 04:14:50PM +0000, Aurelien Aptel wrote:
...
Quoted 5 lines
net/ethtool/ulp_ddp.c new file +0
-0
...
net/ethtool/ulp_ddp.c line 59
Quoted 5 lines
net/ethtool/ulp_ddp.c new file +5
-0+static int ulp_ddp_put_stats64 (struct sk_buff *skb,int attrtype,const u64 *val, +unsigned int count) +{ +struct nlattr *nest; +unsigned int i, attr;
nit: please use reverse xmas tree - longest line to shortest -
for local variable declarations in Networking code.
Re: [PATCH v12 03/26] net/ethtool: add ULP_DDP_{GET,SET} operations for caps and stats
Quoted 5 lines · Simon Horman
net/ethtool/ulp_ddp.c line 59
Quoted 2 lines
net/ethtool/ulp_ddp.c new file +2
-0+struct nlattr *nest; +unsigned int i, attr;nit: please use reverse xmas tree - longest line to shortest -
for local variable declarations in Networking code.
I'll fix it, thanks
PATCH v12 4/26 Documentation: document netlink ULP_DDP_GET/SET messages
In reply to Aurelien AptelAdd detailed documentation about:
- ETHTOOL_MSG_ULP_DDP_GET and ETHTOOL_MSG_ULP_DDP_SET netlink messages
- ETH_SS_ULP_DDP_CAPS and ETH_SS_ULP_DDP_STATS stringsets
ETHTOOL_MSG_ULP_DDP_GET/SET messages are used to configure ULP DDP capabilities and retrieve ULP DDP statistics.
Both statistics and capabilities names can be retrieved dynamically from the kernel via string sets (no need to hardcode them and keep them in sync in ethtool).
--- Documentation/netlink/specs/ethtool.yaml | 102 +++++++++++++++++++ Documentation/networking/ethtool-netlink.rst | 92 +++++++++++++++++ Documentation/networking/statistics.rst | 1 + 3 files changed, 195 insertions(+)
Documentation/netlink/specs/ethtool.yaml +102 -0
@@ -377,6 +377,67 @@ attribute-sets: name: nochange type: nest nested-attributes: bitset + - + name: ulp-ddp-stat + attributes: + - + name: pad + value: 1 + type: pad + - + name: rx-nvmeotcp-sk-add + type: u64 + - + name: rx-nvmeotcp-sk-add-fail + type: u64 + - + name: rx-nvmeotcp-sk-del + type: u64 + - + name: rx-nvmeotcp-ddp-setup + type: u64 + - + name: rx-nvmeotcp-ddp-setup-fail + type: u64 + - + name: rx-nvmeotcp-ddp-teardown + type: u64 + - + name: rx-nvmeotcp-ddp-drop + type: u64 + - + name: rx-nvmeotcp-ddp-resync + type: u64 + - + name: rx-nvmeotcp-ddp-packets + type: u64 + - + name: rx-nvmeotcp-ddp-bytes + type: u64 + - + name: ulp-ddp + attributes: + - + name: header + value: 1 + type: nest + nested-attributes: header + - + name: hw + type: nest + nested-attributes: bitset + - + name: active + type: nest + nested-attributes: bitset + - + name: wanted + type: nest + nested-attributes: bitset + - + name: stats + type: nest + nested-attributes: ulp-ddp-stat - name: channels attributes:
@@ -1692,3 +1753,44 @@ operations: name: mm-ntf doc: Notification for change in MAC Merge configuration. notify: mm-get + - + name: ulp-ddp-get + doc: Get ULP DDP capabilities and stats. + + attribute-set: ulp-ddp + + do: &ulp-ddp-get-op + request: + value: 44 + attributes: + - header + reply: + value: 44 + attributes: + - header + - hw + - active + - stats + dump: *ulp-ddp-get-op + - + name: ulp-ddp-set + doc: Set ULP DDP capabilities. + + attribute-set: ulp-ddp + + do: + request: + value: 45 + attributes: + - header + - wanted + reply: + value: 45 + attributes: + - header + - hw + - active + - + name: ulp-ddp-ntf + doc: Notification for change in ULP DDP capabilities. + notify: ulp-ddp-get
Documentation/networking/ethtool-netlink.rst +92 -0
110 lines not on this page · raw message
Documentation/networking/statistics.rst +1 -0
@@ -172,6 +172,7 @@ statistics are supported in the following commands: - `ETHTOOL_MSG_PAUSE_GET` - `ETHTOOL_MSG_FEC_GET` - `ETHTOOL_MSG_MM_GET` + - `ETHTOOL_MSG_ULP_DDP_GET` debugfs -------
Signature · 1 line
2.34.1
Re: [PATCH v12 04/26] Documentation: document netlink ULP_DDP_GET/SET messages
Quoted 13 lines · Aurelien Aptel
Add detailed documentation about:
-ETHTOOL_MSG_ULP_DDP_GETandETHTOOL_MSG_ULP_DDP_SETnetlink messages
-ETH_SS_ULP_DDP_CAPSandETH_SS_ULP_DDP_STATSstringsetsETHTOOL_MSG_ULP_DDP_GET/SET messages are used to configure ULP DDP capabilities and retrieve ULP DDP statistics.
Both statistics and capabilities names can be retrieved dynamically from the kernel via string sets (no need to hardcode them and keep them in sync in ethtool).
Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>
...
Quoted 4 lines
Documentation/networking/ethtool-netlink.rst +0
-0
...
Quoted 7 lines
Documentation/networking/ethtool-netlink.rst +7
-0+ULP DDP statistics content: + + ===================================================== === =============== + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD`` u64 sockets successfully prepared for offloading + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_ADD_FAIL`` u64 sockets that failed to be prepared for offloading + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_SK_DEL`` u64 sockets where offloading has been removed + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP`` u64 PDUs successfully prepared for Direct Data Placement
Documentation/networking/ethtool-netlink.rst The attributes are propagated to the driver through the following structure: · line 2047
Quoted 10 lines
Documentation/networking/ethtool-netlink.rst +10
-0+ ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_SETUP_FAIL`` u64 PDUs that failed DDP preparation + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DDP_TEARDOWN`` u64 PDUs done with DDP + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_DROP`` u64 PDUs dropped + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_RESYNC`` u64 resync + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_PACKETS`` u64 offloaded PDUs + ``ETHTOOL_A_ULP_DDP_STATS_RX_NVMEOTCP_BYTES`` u64 offloaded bytes + ===================================================== === =============== + +The names of each statistics are global. They can be retrieved via the +``ETH_SS_ULP_DDP_STATS`` string set.
'make htmldocs' seems a bit unhappy about the table above.
.../ethtool-netlink.rst:2038: WARNING: Malformed table. Text in column margin in table line 6.
Re: [PATCH v12 04/26] Documentation: document netlink ULP_DDP_GET/SET messages
Quoted 4 lines · Simon Horman
'make htmldocs' seems a bit unhappy about the table above.
.../ethtool-netlink.rst:2038: WARNING: Malformed table. Text in column margin in table line 6.
Turns out the column separator is 2 spaces.
I'll fix it, thanks.
PATCH v12 5/26 iov_iter: skip copy if src == dst for direct data placement
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
When using direct data placement (DDP) the NIC could write the payload directly into the destination buffer and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination buffer we check if (src == dst), and skip the copy when it's true.
--- lib/iov_iter.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
lib/iov_iter.c +7 -1
@@ -313,9 +313,15 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)return 0 ;if (user_backed_iter (i))might_fault (); +/* +* When using direct data placement (DDP) the hardware writes +* data directly to the destination buffer, and constructs +* IOVs such that they point to this data. +* Thus, when the src == dst we skip the memcpy. +*/ iterate_and_advance (i, bytes, base, len, off,copyout (base, addr + off, len), -memcpy (base, addr + off, len) + (base != addr + off) &&memcpy (base, addr + off, len) )return bytes;
Signature · 1 line
2.34.1
Re: [PATCH v12 05/26] iov_iter: skip copy if src == dst for direct data placement
Quoted 30 lines · Aurelien Aptel
From: Ben Ben-Ishay <[email protected]>
When using direct data placement (DDP) the NIC could write the payload directly into the destination buffer and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination buffer we check if (src == dst), and skip the copy when it's true.
Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>--- lib/iov_iter.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)lib/iov_iter.c +3
-0@@ -313,9 +313,15 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)return 0 ;if (user_backed_iter (i))might_fault (); +/* +* When using direct data placement (DDP) the hardware writes +* data directly to the destination buffer, and constructs
lib/iov_iter.c size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) · line 327
Quoted 10 lines
lib/iov_iter.c +4
-1+* IOVs such that they point to this data. +* Thus, when the src == dst we skip the memcpy. +*/ iterate_and_advance (i, bytes, base, len, off,copyout (base, addr + off, len), -memcpy (base, addr + off, len) + (base != addr + off) &&memcpy (base, addr + off, len) )return bytes;
Looks good,
PATCH v12 6/26 net/tls,core: export get_netdev_for_sock
In reply to Aurelien Aptel* remove netdev_sk_get_lowest_dev() from net/core
* move get_netdev_for_sock() from net/tls to net/core
get_netdev_for_sock() is a utility that is used to obtain the net_device structure from a connected socket.
Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.
--- include/linux/netdevice.h | 3 +-- net/core/dev.c | 26 +++++++++++++------------- net/tls/tls_device.c | 16 ---------------- 3 files changed, 14 insertions(+), 31 deletions(-)
include/linux/netdevice.h +1 -2
@@ -3135,8 +3135,7 @@ int init_dummy_netdev(struct net_device *dev);struct net_device *netdev_get_xmit_slave (struct net_device *dev,struct sk_buff *skb,bool all_slaves); -struct net_device *netdev_sk_get_lowest_dev (struct net_device *dev, -struct sock *sk); +struct net_device *get_netdev_for_sock (struct sock *sk);struct net_device *dev_get_by_index (struct net *net,int ifindex);struct net_device *__dev_get_by_index (struct net *net,int ifindex); struct net_device *netdev_get_by_index(struct net *net, int ifindex,
net/core/dev.c +13 -13
@@ -8151,27 +8151,27 @@ static struct net_device *netdev_sk_get_lower_dev(struct net_device *dev, }/** -* netdev_sk_get_lowest_dev - Get the lowest device in chain given device and socket -* @dev: device +* get_netdev_for_sock - Get the lowest device in socket * @sk: the socket * -* %NULL is returned if no lower device is found. +* Assumes that the socket is already connected. +* Returns the lower device or %NULL if no lower device is found. */ - -struct net_device *netdev_sk_get_lowest_dev (struct net_device *dev, -struct sock *sk) +struct net_device *get_netdev_for_sock (struct sock *sk) { -struct net_device *lower; +struct dst_entry *dst =sk_dst_get (sk); +struct net_device *dev, *lower; - lower =netdev_sk_get_lower_dev (dev, sk); -while (lower) { +if (unlikely (!dst)) +return NULL ; + dev = dst->dev; +while ((lower =netdev_sk_get_lower_dev (dev, sk))) dev = lower; - lower =netdev_sk_get_lower_dev (dev, sk); - } - +dev_hold (dev); +dst_release (dst);return dev; } -EXPORT_SYMBOL (netdev_sk_get_lowest_dev); +EXPORT_SYMBOL_GPL (get_netdev_for_sock);static void netdev_adjacent_add_links (struct net_device *dev) {
net/tls/tls_device.c +0 -16
@@ -120,22 +120,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx)tls_device_free_ctx (ctx); } -/* We assume that the socket is already connected */ -static struct net_device *get_netdev_for_sock (struct sock *sk) -{ -struct dst_entry *dst =sk_dst_get (sk); -struct net_device *netdev =NULL ; - -if (likely (dst)) { - netdev =netdev_sk_get_lowest_dev (dst->dev, sk); -dev_hold (netdev); - } - -dst_release (dst); - -return netdev; -} -static void destroy_record (struct tls_record_info *record) {int i;
Signature · 1 line
2.34.1
PATCH v12 7/26 nvme-tcp: Add DDP offload control path
In reply to Aurelien AptelFrom: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via ulp_ddp_limits.
A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on NETDEV_GOING_DOWN events to allow the device to go down in a follow-up patch.
--- drivers/nvme/host/tcp.c | 272 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 263 insertions(+), 9 deletions(-)
drivers/nvme/host/tcp.c +263 -9
@@ -20,6 +20,10 @@#include <net/busy_poll.h> #include <trace/events/sock.h> +#ifdef CONFIG_ULP_DDP +#include <net/ulp_ddp.h> +#endif +#include "nvme.h" #include "fabrics.h"
@@ -118,6 +122,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFF_DDP = 3, }; enum nvme_tcp_recv_state {
@@ -145,6 +150,19 @@ struct nvme_tcp_queue {size_t ddgst_remaining;unsigned int nr_cqe; +#ifdef CONFIG_ULP_DDP +/* +* resync_req is a speculative PDU header tcp seq number (with +* an additional flag at 32 lower bits) that the HW send to +* the SW, for the SW to verify. +* - The 32 high bits store the seq number +* - The 32 low bits are used as a flag to know if a request +* is pending (ULP_DDP_RESYNC_PENDING). +*/ +atomic64_t resync_req; +struct ulp_ddp_limits ddp_limits; +#endif +/* send state */ struct nvme_tcp_request *request;
@@ -187,6 +205,9 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; +u32 offload_io_threshold; };static LIST_HEAD (nvme_tcp_ctrl_list);
@@ -290,6 +311,204 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; } +#ifdef CONFIG_ULP_DDP + +static bool nvme_tcp_ddp_query_limits (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +int ret; + +if (!netdev->netdev_ops->ulp_ddp_ops->limits) +return false; + + queue->ddp_limits.type =ULP_DDP_NVME ; + ret = netdev->netdev_ops->ulp_ddp_ops->limits (netdev, &queue->ddp_limits); +if (ret == -EOPNOTSUPP ) { +return false; + }else if (ret) { +WARN_ONCE (ret,"ddp limits failed (ret=%d)" , ret); +return false; + } + +dev_dbg_ratelimited (queue->ctrl->ctrl.device, +"netdev %s offload limits: max_ddp_sgl_len %d\n" , + netdev->name, queue->ddp_limits.max_ddp_sgl_len); + +return true; +} + +static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +if (!netdev || !queue) +return false; + +/* If we cannot query the netdev limitations, do not offload */ +if (!nvme_tcp_ddp_query_limits (netdev, queue)) +return false; + +/* If netdev supports nvme-tcp ddp offload, we can offload */ +if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active)) +return true; + +return false; +} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { + .resync_request = nvme_tcp_resync_request, +}; + +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +int ret; + + config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ; + config.nvmeotcp.cpda =0 ; + config.nvmeotcp.dgst = + queue->hdr_digest ?NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.nvmeotcp.dgst |= + queue->data_digest ?NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.nvmeotcp.queue_size = queue->ctrl->ctrl.sqsize +1 ; + config.nvmeotcp.queue_id =nvme_tcp_queue_id (queue); + config.nvmeotcp.io_cpu = queue->io_cpu; + +/* Socket ops keep a netdev reference. It is put in +* nvme_tcp_unoffload_socket(). This ref is dropped on +* NETDEV_GOING_DOWN events to allow the device to go down +*/ +dev_hold (netdev); + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add (netdev, + queue->sock->sk, + &config); +if (ret) { +dev_put (netdev); +return ret; + } + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; +set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +return 0 ; +} + +static void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (!netdev) { +dev_info_ratelimited (queue->ctrl->ctrl.device,"netdev not found\n" ); +return ; + } + +clear_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); + + netdev->netdev_ops->ulp_ddp_ops->sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ; +dev_put (netdev);/* held by offload_socket */ +} + +static void nvme_tcp_offload_apply_limits (struct nvme_tcp_queue *queue, +struct net_device *netdev) +{ + queue->ctrl->offloading_netdev = netdev; + queue->ctrl->ctrl.max_segments = queue->ddp_limits.max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + queue->ddp_limits.max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 ); + queue->ctrl->offload_io_threshold = queue->ddp_limits.io_threshold; + +/* offloading HW doesn't support full ccid range, apply the quirk */ + queue->ctrl->ctrl.quirks |= + queue->ddp_limits.nvmeotcp.full_ccid_range ?0 :NVME_QUIRK_SKIP_CID_GEN ; +} + +/* In presence of packet drops or network packet reordering, the device may lose +* synchronization between the TCP stream and the L5P framing, and require a +* resync with the kernel's TCP stack. +* +* - NIC HW identifies a PDU header at some TCP sequence number, +* and asks NVMe-TCP to confirm it. +* - When NVMe-TCP observes the requested TCP sequence, it will compare +* it with the PDU header TCP sequence, and report the result to the +* NIC driver +*/ +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{ +u64 pdu_seq =TCP_SKB_CB (skb)->seq + offset - queue->pdu_offset; +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 pdu_val = (pdu_seq <<32 ) |ULP_DDP_RESYNC_PENDING ; +u64 resync_val; +u32 resync_seq; + + resync_val =atomic64_read (&queue->resync_req); +/* Lower 32 bit flags. Check validity of the request */ +if ((resync_val &ULP_DDP_RESYNC_PENDING ) ==0 ) +return ; + +/* +* Obtain and check requested sequence number: is this PDU header +* before the request? +*/ + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ; + +/* +* The atomic operation guarantees that we don't miss any NIC driver +* resync requests submitted after the above checks. +*/ +if (atomic64_cmpxchg (&queue->resync_req, pdu_val, + pdu_val & ~ULP_DDP_RESYNC_PENDING ) != +atomic64_read (&queue->resync_req)) + netdev->netdev_ops->ulp_ddp_ops->resync (netdev, + queue->sock->sk, + pdu_seq); +} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +/* +* "seq" (TCP seq number) is what the HW assumes is the +* beginning of a PDU. The nvme-tcp layer needs to store the +* number along with the "flags" (ULP_DDP_RESYNC_PENDING) to +* indicate that a request is pending. +*/ +atomic64_set (&queue->resync_req, (((uint64_t )seq <<32 ) | flags)); + +return true; +} + +#else + +static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +return false; +} + +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +return -EOPNOTSUPP ; +} + +static void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{} + +static void nvme_tcp_offload_apply_limits (struct nvme_tcp_queue *queue, +struct net_device *netdev) +{} + +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{} + +#endif +static void nvme_tcp_init_iter (struct nvme_tcp_request *req,unsigned int dir) {
@@ -732,6 +951,9 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +nvme_tcp_resync_response (queue, skb, *offset); + ret =skb_copy_bits (skb, *offset, &pdu[queue->pdu_offset], rcv_len);if (unlikely (ret))
@@ -1795,6 +2017,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_ops (queue);cancel_work_sync (&queue->io_work); +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +nvme_tcp_unoffload_socket (queue); }static void nvme_tcp_stop_queue (struct nvme_ctrl *nctrl,int qid)
@@ -1831,25 +2055,55 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[idx]; +struct net_device *netdev;int ret; queue->rd_enabled = true;nvme_tcp_init_recv_ctx (queue);nvme_tcp_setup_sock_ops (queue); -if (idx) +if (idx) { ret =nvmf_connect_io_queue (nctrl, idx); -else +if (ret) + goto err; + + netdev = queue->ctrl->offloading_netdev; +if (is_netdev_ulp_offload_active (netdev, queue)) { + ret =nvme_tcp_offload_socket (queue); +if (ret) { +dev_info (nctrl->device, +"failed to setup offload on queue %d ret=%d\n" , + idx, ret); + } + } + }else { ret =nvmf_connect_admin_queue (nctrl); +if (ret) + goto err; -if (!ret) { -set_bit (NVME_TCP_Q_LIVE , &queue->flags); - }else { -if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) -__nvme_tcp_stop_queue (queue); -dev_err (nctrl->device, -"failed to connect queue: %d ret=%d\n" , idx, ret); + netdev =get_netdev_for_sock (queue->sock->sk); +if (!netdev) { +dev_info_ratelimited (ctrl->ctrl.device,"netdev not found\n" ); + ctrl->offloading_netdev =NULL ; + goto done; + } +if (is_netdev_ulp_offload_active (netdev, queue)) +nvme_tcp_offload_apply_limits (queue, netdev); +/* +* release the device as no offload context is +* established yet. +*/ +dev_put (netdev); } + +done: +set_bit (NVME_TCP_Q_LIVE , &queue->flags); +return 0 ; +err: +if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) +__nvme_tcp_stop_queue (queue); +dev_err (nctrl->device, +"failed to connect queue: %d ret=%d\n" , idx, ret);return ret; }
Signature · 1 line
2.34.1
Re: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 40 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
ulp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on
NETDEV_GOING_DOWNevents to allow the device to go down in a follow-up patch.
Quoted 8 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>---
For NVMe related code :-
Offload feature is configurable and maybe not be turned on in the absence of the H/W. In order to keep the nvme/host/tcp.c file small to only handle core related functionality, I wonder if we should to move tcp-offload code into it's own file say nvme/host/tcp-offload.c ?
-ck
Sagi Grimberg
maintainerRe: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 50 lines · Chaitanya Kulkarni
Quoted 40 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
ulp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on
NETDEV_GOING_DOWNevents to allow the device to go down in a follow-up patch.Quoted 8 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>---
Quoted 6 lines
For NVMe related code :-
Offload feature is configurable and maybe not be turned on in the absence of the H/W. In order to keep the
nvme/host/tcp.cfile small to only handle core related functionality, I wonder if we should to move tcp-offload code into it's own file saynvme/host/tcp-offload.c?
Maybe. it wouldn't be tcp_offload.c but rather tcp_ddp.c because its not offloading the tcp stack but rather doing direct data placement.
If we are going to do that it will pollute nvme.h or add a common header file, which is something I'd like to avoid if possible.
Re: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 62 lines · Sagi Grimberg
Quoted 50 lines · Chaitanya Kulkarni
Quoted 40 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
ulp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on
NETDEV_GOING_DOWNevents to allow the device to go down in a follow-up patch.Quoted 8 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>---
Quoted 8 lines
For NVMe related code :-
Offload feature is configurable and maybe not be turned on in the absence of the H/W. In order to keep the
nvme/host/tcp.cfile small to only handle
core related functionality, I wonder if we should to move tcp-offload code
into it's own file saynvme/host/tcp-offload.c?
Quoted 3 lines
Maybe. it wouldn't be
tcp_offload.cbut rathertcp_ddp.cbecause its not offloading the tcp stack but rather doing direct data placement.
fine ...
Quoted 2 lines
If we are going to do that it will pollute
nvme.hor add a common header file, which is something I'd like to avoid if possible.
my comment was mainly based on how we separated the core code from configurable features, and wondering any decision we make for host will also apply for the target ddp code in future, but you prefer to keep it as it is let's not bloat header ...
-ck
Re: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Hi Sagi,
Quoted 63 lines · Chaitanya Kulkarni
Quoted 62 lines · Sagi Grimberg
Quoted 58 lines · Chaitanya Kulkarni
Quoted 40 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
ulp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on
NETDEV_GOING_DOWNevents to allow the device to go down in a follow-up patch.Quoted 8 lines
Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>---
For NVMe related code :-
Offload feature is configurable and maybe not be turned on in the absence of the H/W. In order to keep the
nvme/host/tcp.cfile small to only handle
core related functionality, I wonder if we should to move tcp-offload code
into it's own file saynvme/host/tcp-offload.c?
Quoted 8 lines
Quoted 3 lines
Maybe. it wouldn't be
tcp_offload.cbut rathertcp_ddp.cbecause its not offloading the tcp stack but rather doing direct data placement.fine ...
Quoted 2 lines
If we are going to do that it will pollute
nvme.hor add a common header file, which is something I'd like to avoid if possible.
Would you like us to try doing so and see how will it look like ?
I actually think that it will be easier to maintain if we split it to tcp_ddp.c (+ optional adding of tcp.h under driver/nvme/host...)
Quoted 10 lines
my comment was mainly based on how we separated the core code from configurable features, and wondering any decision we make for host will also apply for the target ddp code in future, but you prefer to keep it as it is let's not bloat header ...
-ck
Sagi Grimberg
maintainerRe: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
In reply to Aurelien AptelHey,
Sorry for the late review on this, I've been struggling with prioritizing this due to lack of time.
I'll most likely review in batches.
Quoted 95 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding thenvme_tcp_queuethat is used by this routine. Finally, the request is recorded in thenvme_tcp_queue.3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (resync), which will update the HW, and resume offload when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1 for queue of size N) where the linux nvme driver uses part of the 16 bit CCID for generation counter. To address that, we use the existing quirk in the nvme layer when the HW driver advertises if the device is not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via
ulp_ddp_limits.A follow-up patch introduces the data-path changes required for this offload.
Socket operations need a netdev reference. This reference is dropped on
NETDEV_GOING_DOWNevents to allow the device to go down in a follow-up patch.Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>--- drivers/nvme/host/tcp.c | 272 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 263 insertions(+), 9 deletions(-)drivers/nvme/host/tcp.c +18
-0@@ -20,6 +20,10 @@#include <net/busy_poll.h> #include <trace/events/sock.h> +#ifdef CONFIG_ULP_DDP +#include <net/ulp_ddp.h> +#endif +#include "nvme.h" #include "fabrics.h" @@ -118,6 +122,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_ALLOCATED =0 ,NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 , + NVME_TCP_Q_OFF_DDP = 3, }; enum nvme_tcp_recv_state {@@ -145,6 +150,19 @@ struct nvme_tcp_queue {size_t ddgst_remaining;unsigned int nr_cqe; +#ifdef CONFIG_ULP_DDP +/* +* resync_req is a speculative PDU header tcp seq number (with +* an additional flag at 32 lower bits) that the HW send to +* the SW, for the SW to verify. +* - The 32 high bits store the seq number +* - The 32 low bits are used as a flag to know if a request +* is pending (ULP_DDP_RESYNC_PENDING). +*/ +atomic64_t resync_req; +struct ulp_ddp_limits ddp_limits; +#endif +/* send state */ struct nvme_tcp_request *request;
drivers/nvme/host/tcp.c struct nvme_tcp_ctrl { · line 210
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0@@ -187,6 +205,9 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; +u32 offload_io_threshold;
ddp_netdevddp_io_threashold
Quoted 6 lines
drivers/nvme/host/tcp.c +0
-0};static LIST_HEAD (nvme_tcp_ctrl_list);@@ -290,6 +311,204 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,return nvme_tcp_pdu_data_left (req) <= len; }
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 322
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+#ifdef CONFIG_ULP_DDP + +static bool nvme_tcp_ddp_query_limits (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +int ret; + +if (!netdev->netdev_ops->ulp_ddp_ops->limits) +return false;
Can we expose all of these ops in wrappers ala:
netdev_ulp_ddp_limits(netdev, &limits) netdev_ulp_ddp_sk_add(netdev, sk, &nvme_tcp_ddp_ulp_ops) netdev_ulp_ddp_sk_del(netdev, sk) netdev_ulp_ddp_resync(netdev, skb, seq)
etc...
Quoted 12 lines
drivers/nvme/host/tcp.c +12
-0+ + queue->ddp_limits.type =ULP_DDP_NVME ; + ret = netdev->netdev_ops->ulp_ddp_ops->limits (netdev, &queue->ddp_limits); +if (ret == -EOPNOTSUPP ) { +return false; + }else if (ret) { +WARN_ONCE (ret,"ddp limits failed (ret=%d)" , ret); +return false; + } + + dev_dbg_ratelimited(queue->ctrl->ctrl.device, + "netdev %s offload limits: max_ddp_sgl_len %d\n",
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 344
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ netdev->name, queue->ddp_limits.max_ddp_sgl_len); + +return true; +} + +static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +if (!netdev || !queue) +return false;
Is it reasonable to be called here with !netdev or !queue ?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 352
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ +/* If we cannot query the netdev limitations, do not offload */ +if (!nvme_tcp_ddp_query_limits (netdev, queue)) +return false; + +/* If netdev supports nvme-tcp ddp offload, we can offload */ +if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active)) +return true;
This should be coming from the API itself, have the limits query api fail if this is off.
btw, what is the active thing? is this driven from ethtool enable? what happens if the user disables it while there is a ulp using it?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 354
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+ +return false;
This can be folded to the above function.
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 368
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ .resync_request = nvme_tcp_resync_request, +}; + +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +int ret; + + config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ;
Question, what happens if the pfv changes, is the ddp guaranteed to work?
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ config.nvmeotcp.cpda =0 ; + config.nvmeotcp.dgst = + queue->hdr_digest ?NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.nvmeotcp.dgst |= + queue->data_digest ?NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.nvmeotcp.queue_size = queue->ctrl->ctrl.sqsize +1 ; + config.nvmeotcp.queue_id =nvme_tcp_queue_id (queue);
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 385
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ config.nvmeotcp.io_cpu = queue->io_cpu; + +/* Socket ops keep a netdev reference. It is put in +* nvme_tcp_unoffload_socket(). This ref is dropped on +* NETDEV_GOING_DOWN events to allow the device to go down +*/ +dev_hold (netdev); + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add (netdev, + queue->sock->sk, + &config);
It would be preferred if dev_hold would be taken in sk_add and released in sk_del so that the ulp does not need to worry acount it.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 391
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+if (ret) { +dev_put (netdev); +return ret; + } + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops;
can also be folded inside an api.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 403
Quoted 12 lines
drivers/nvme/host/tcp.c +12
-0+set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +return 0 ; +} + +static void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (!netdev) { +dev_info_ratelimited (queue->ctrl->ctrl.device,"netdev not found\n" ); +return ; + }
Again, is it reasonable to be called here with !netdev?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 410
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ +clear_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); + + netdev->netdev_ops->ulp_ddp_ops->sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ; +dev_put (netdev);/* held by offload_socket */
Both can be done by the api instead of the ulp itself.
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 419
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+} + +static void nvme_tcp_offload_apply_limits (struct nvme_tcp_queue *queue, +struct net_device *netdev) +{ + queue->ctrl->offloading_netdev = netdev; + queue->ctrl->ctrl.max_segments = queue->ddp_limits.max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + queue->ddp_limits.max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 );
this is SECTOR_SHIFT?
Quoted 39 lines
drivers/nvme/host/tcp.c +39
-0+ queue->ctrl->offload_io_threshold = queue->ddp_limits.io_threshold; + +/* offloading HW doesn't support full ccid range, apply the quirk */ + queue->ctrl->ctrl.quirks |= + queue->ddp_limits.nvmeotcp.full_ccid_range ?0 :NVME_QUIRK_SKIP_CID_GEN ; +} + +/* In presence of packet drops or network packet reordering, the device may lose +* synchronization between the TCP stream and the L5P framing, and require a +* resync with the kernel's TCP stack. +* +* - NIC HW identifies a PDU header at some TCP sequence number, +* and asks NVMe-TCP to confirm it. +* - When NVMe-TCP observes the requested TCP sequence, it will compare +* it with the PDU header TCP sequence, and report the result to the +* NIC driver +*/ +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{ +u64 pdu_seq =TCP_SKB_CB (skb)->seq + offset - queue->pdu_offset; +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 pdu_val = (pdu_seq <<32 ) |ULP_DDP_RESYNC_PENDING ; +u64 resync_val; +u32 resync_seq; + + resync_val =atomic64_read (&queue->resync_req); +/* Lower 32 bit flags. Check validity of the request */ +if ((resync_val &ULP_DDP_RESYNC_PENDING ) ==0 ) +return ; + +/* +* Obtain and check requested sequence number: is this PDU header +* before the request? +*/ + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ; +
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 468
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+/* +* The atomic operation guarantees that we don't miss any NIC driver +* resync requests submitted after the above checks. +*/ +if (atomic64_cmpxchg (&queue->resync_req, pdu_val, + pdu_val & ~ULP_DDP_RESYNC_PENDING ) != +atomic64_read (&queue->resync_req)) + netdev->netdev_ops->ulp_ddp_ops->resync (netdev, + queue->sock->sk, + pdu_seq);
Who else is doing an atomic on this value? and what happens if the cmpxchg fails?
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 481
Quoted 13 lines
drivers/nvme/host/tcp.c +13
-0+} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +/* +* "seq" (TCP seq number) is what the HW assumes is the +* beginning of a PDU. The nvme-tcp layer needs to store the +* number along with the "flags" (ULP_DDP_RESYNC_PENDING) to +* indicate that a request is pending. +*/ +atomic64_set (&queue->resync_req, (((uint64_t )seq <<32 ) | flags));
Question, is this coming from multiple contexts? what contexts are competing here that make it an atomic operation? It is unclear what is going on here tbh.
Quoted 62 lines
drivers/nvme/host/tcp.c +36
-0+ + return true; +} + +#else + +static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +return false; +} + +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +return -EOPNOTSUPP ; +} + +static void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{} + +static void nvme_tcp_offload_apply_limits (struct nvme_tcp_queue *queue, +struct net_device *netdev) +{} + +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{} + +#endif +static void nvme_tcp_init_iter (struct nvme_tcp_request *req,unsigned int dir) {@@ -732,6 +951,9 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +nvme_tcp_resync_response (queue, skb, *offset); + ret =skb_copy_bits (skb, *offset, &pdu[queue->pdu_offset], rcv_len);if (unlikely (ret))@@ -1795,6 +2017,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_ops (queue);cancel_work_sync (&queue->io_work); +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +nvme_tcp_unoffload_socket (queue); }static void nvme_tcp_stop_queue (struct nvme_ctrl *nctrl,int qid)@@ -1831,25 +2055,55 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[idx]; +struct net_device *netdev;int ret; queue->rd_enabled = true;nvme_tcp_init_recv_ctx (queue);nvme_tcp_setup_sock_ops (queue);
drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2071
Quoted 10 lines
drivers/nvme/host/tcp.c +6
-2-if (idx) +if (idx) { ret =nvmf_connect_io_queue (nctrl, idx); -else +if (ret) + goto err; + + netdev = queue->ctrl->offloading_netdev; +if (is_netdev_ulp_offload_active (netdev, queue)) {
Seems redundant to pass netdev as an argument here.
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+ ret =nvme_tcp_offload_socket (queue); +if (ret) { +dev_info (nctrl->device, +"failed to setup offload on queue %d ret=%d\n" , + idx, ret); + } + } + }else { ret =nvmf_connect_admin_queue (nctrl); +if (ret)
drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2084
Quoted 10 lines
drivers/nvme/host/tcp.c +2
-7+ goto err; -if (!ret) { -set_bit (NVME_TCP_Q_LIVE , &queue->flags); - }else { -if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) -__nvme_tcp_stop_queue (queue); -dev_err (nctrl->device, -"failed to connect queue: %d ret=%d\n" , idx, ret); + netdev =get_netdev_for_sock (queue->sock->sk);
Is there any chance that this is a different netdev than what is already recorded? doesn't make sense to me.
drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2096
Quoted 12 lines
drivers/nvme/host/tcp.c +12
-0+if (!netdev) { +dev_info_ratelimited (ctrl->ctrl.device,"netdev not found\n" ); + ctrl->offloading_netdev =NULL ; + goto done; + } +if (is_netdev_ulp_offload_active (netdev, queue)) +nvme_tcp_offload_apply_limits (queue, netdev); +/* +* release the device as no offload context is +* established yet. +*/ +dev_put (netdev);
the put is unclear, what does it pair with? the get_netdev_for_sock?
Quoted 13 lines
drivers/nvme/host/tcp.c +9
-0} + +done: +set_bit (NVME_TCP_Q_LIVE , &queue->flags); +return 0 ; +err: +if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) +__nvme_tcp_stop_queue (queue); +dev_err (nctrl->device, +"failed to connect queue: %d ret=%d\n" , idx, ret);return ret; }
Re: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 10 lines · Sagi Grimberg
drivers/nvme/host/tcp.c struct nvme_tcp_ctrl { · line 210
Quoted 7 lines
drivers/nvme/host/tcp.c +3
-0@@ -187,6 +205,9 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; + +struct net_device *offloading_netdev; +u32 offload_io_threshold;
ddp_netdevddp_io_threashold
Sure, will change it.
Quoted 10 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 322
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+static bool nvme_tcp_ddp_query_limits (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +int ret; + +if (!netdev->netdev_ops->ulp_ddp_ops->limits) +return false;Can we expose all of these ops in wrappers ala:
Quoted 6 lines
netdev_ulp_ddp_limits(netdev, &limits)netdev_ulp_ddp_sk_add(netdev, sk, &nvme_tcp_ddp_ulp_ops)netdev_ulp_ddp_sk_del(netdev, sk)netdev_ulp_ddp_resync(netdev, skb, seq)etc...
Sure, we will add simple wrappers in ulp_ddp.h to check for the function pointers.
Quoted 7 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 344
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +if (!netdev || !queue) +return false;Is it reasonable to be called here with !netdev or !queue ?
The check is needed only for the IO queue case but we can move it earlier in nvme_tcp_start_queue().
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 352
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ +/* If we cannot query the netdev limitations, do not offload */ +if (!nvme_tcp_ddp_query_limits (netdev, queue)) +return false; + +/* If netdev supports nvme-tcp ddp offload, we can offload */ +if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active)) +return true;This should be coming from the API itself, have the limits query api fail if this is off.
We can move the function to the ULP DDP layer.
Quoted 2 lines
btw, what is the active thing? is this driven from ethtool enable? what happens if the user disables it while there is a ulp using it?
The active bits are indeed driven by ethtool according to the design Jakub suggested.
The nvme-tcp connection will have to be reconnected to see the effect of changing the bit.
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + return false;This can be folded to the above function.
We won't be able to check for TLS in a common wrapper. We think this should be kept.
Quoted 10 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 368
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +int ret; + + config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ;Question, what happens if the pfv changes, is the ddp guaranteed to work?
The existing HW supports only NVME_TCP_PFV_1_0.
Once a new version will be used, the device driver should fail the sk_add().
Quoted 18 lines
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ config.nvmeotcp.cpda =0 ; + config.nvmeotcp.dgst = + queue->hdr_digest ?NVME_TCP_HDR_DIGEST_ENABLE :0 ; + config.nvmeotcp.dgst |= + queue->data_digest ?NVME_TCP_DATA_DIGEST_ENABLE :0 ; + config.nvmeotcp.queue_size = queue->ctrl->ctrl.sqsize +1 ; + config.nvmeotcp.queue_id =nvme_tcp_queue_id (queue);drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 385
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ config.nvmeotcp.io_cpu = queue->io_cpu; + +/* Socket ops keep a netdev reference. It is put in +* nvme_tcp_unoffload_socket(). This ref is dropped on +* NETDEV_GOING_DOWN events to allow the device to go down +*/ +dev_hold (netdev); + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add (netdev, + queue->sock->sk, + &config);
Quoted 3 lines
It would be preferred if
dev_holdwould be taken insk_addand released insk_delso that the ulp does not need to worry acount it.
Sure, we will move the refcount accounting to the ulp ddp wrapper.
Quoted 2 lines
Quoted 1 line
+ inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops =
&nvme_tcp_ddp_ulp_ops;can also be folded inside an api.
Sure, we will move this to sk_add() and add a parameter for the ops.
Quoted 10 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 403
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+static void nvme_tcp_unoffload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; + +if (!netdev) { +dev_info_ratelimited (queue->ctrl->ctrl.device,"netdev not found\n" ); +return ; + }Again, is it reasonable to be called here with !netdev?
No it's redundant, we will remove it.
Quoted 9 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 410
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ +clear_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); + + netdev->netdev_ops->ulp_ddp_ops->sk_del (netdev, queue->sock->sk); + +inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops =NULL ; +dev_put (netdev);/* held by offload_socket */ Both can be done by the api instead of the ulp itself.
Sure, we will move it.
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 419
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+} + +static void nvme_tcp_offload_apply_limits (struct nvme_tcp_queue *queue, +struct net_device *netdev) +{ + queue->ctrl->offloading_netdev = netdev; + queue->ctrl->ctrl.max_segments = queue->ddp_limits.max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + queue->ddp_limits.max_ddp_sgl_len << (ilog2 (SZ_4K ) -9 );this is
SECTOR_SHIFT?
Yes it is, we will use it.
Quoted 37 lines
Quoted 37 lines
drivers/nvme/host/tcp.c +37
-0+/* In presence of packet drops or network packet reordering, the device may lose +* synchronization between the TCP stream and the L5P framing, and require a +* resync with the kernel's TCP stack. +* +* - NIC HW identifies a PDU header at some TCP sequence number, +* and asks NVMe-TCP to confirm it. +* - When NVMe-TCP observes the requested TCP sequence, it will compare +* it with the PDU header TCP sequence, and report the result to the +* NIC driver +*/ +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{ +u64 pdu_seq =TCP_SKB_CB (skb)->seq + offset - queue->pdu_offset; +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 pdu_val = (pdu_seq <<32 ) |ULP_DDP_RESYNC_PENDING ; +u64 resync_val; +u32 resync_seq; + + resync_val =atomic64_read (&queue->resync_req); +/* Lower 32 bit flags. Check validity of the request */ +if ((resync_val &ULP_DDP_RESYNC_PENDING ) ==0 ) +return ; + +/* +* Obtain and check requested sequence number: is this PDU header +* before the request? +*/ + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ; + +/* +* The atomic operation guarantees that we don't miss any NIC driver +* resync requests submitted after the above checks. +*/ + if (atomic64_cmpxchg(&queue->resync_req, pdu_val,
Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 468
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ pdu_val & ~ULP_DDP_RESYNC_PENDING) != + atomic64_read(&queue->resync_req)) + netdev->netdev_ops->ulp_ddp_ops->resync(netdev, + queue->sock->sk, + pdu_seq);Who else is doing an atomic on this value? and what happens if the cmpxchg fails?
The driver thread can set queue->resync_req concurrently in patch "net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload" in function nvmeotcp_update_resync().
If the cmpxchg fails it means a new resync request was triggered by the HW, the old request will be dropped and the new one will be processed by a later PDU.
Quoted 9 lines
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +/* +* "seq" (TCP seq number) is what the HW assumes is the +* beginning of a PDU. The nvme-tcp layer needs to store the
Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 481
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+* number along with the "flags" (ULP_DDP_RESYNC_PENDING) to +* indicate that a request is pending. +*/ +atomic64_set (&queue->resync_req, (((uint64_t )seq <<32 ) | flags));Question, is this coming from multiple contexts? what contexts are competing here that make it an atomic operation? It is unclear what is going on here tbh.
The driver could get a resync request and set queue->resync_req concurrently while processing HW CQEs as you can see in patch "net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload" in function nvmeotcp_update_resync().
The resync flow is:
nvme-tcp mlx5 hw
| | |
| | sends CQE with
| | resync request
| | <----------------------'
| nvmeotcp_update_resync()
nvme_tcp_resync_request() <-----------'|
we store the request
Later, while receiving PDUs we check for pending requests. If there is one, we send call nvme_tcp_resync_response() which calls into mlx5 to send the response to the HW.
Quoted 11 lines
Quoted 11 lines
drivers/nvme/host/tcp.c +1
-0@@ -1831,25 +2055,55 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[idx]; +struct net_device *netdev;int ret; queue->rd_enabled = true;nvme_tcp_init_recv_ctx (queue);nvme_tcp_setup_sock_ops (queue);
Quoted 11 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2071
Quoted 9 lines
drivers/nvme/host/tcp.c +6
-2-if (idx) +if (idx) { ret =nvmf_connect_io_queue (nctrl, idx); -else +if (ret) + goto err; + + netdev = queue->ctrl->offloading_netdev; +if (is_netdev_ulp_offload_active (netdev, queue)) {Seems redundant to pass netdev as an argument here.
Thanks, we will remove it.
Quoted 21 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+ ret =nvme_tcp_offload_socket (queue); +if (ret) { +dev_info (nctrl->device, +"failed to setup offload on queue %d ret=%d\n" , + idx, ret); + } + } + }else { ret =nvmf_connect_admin_queue (nctrl); +if (ret)drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2084
Quoted 10 lines
drivers/nvme/host/tcp.c +2
-7+ goto err; -if (!ret) { -set_bit (NVME_TCP_Q_LIVE , &queue->flags); - }else { -if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) -__nvme_tcp_stop_queue (queue); -dev_err (nctrl->device, -"failed to connect queue: %d ret=%d\n" , idx, ret); + netdev =get_netdev_for_sock (queue->sock->sk);
Quoted 2 lines
Is there any chance that this is a different netdev than what is already recorded? doesn't make sense to me.
The idea is that we are first starting the admin queue, which looks up the netdev associated with the socket and stored in the queue. Later, when the IO queues are started, we use the recorded netdev.
In cases of bonding or vlan, a netdev can have lower device links, which get_netdev_for_sock() will look up.
Do you see any gap?
Quoted 13 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2096
Quoted 12 lines
drivers/nvme/host/tcp.c +12
-0+if (!netdev) { +dev_info_ratelimited (ctrl->ctrl.device,"netdev not found\n" ); + ctrl->offloading_netdev =NULL ; + goto done; + } +if (is_netdev_ulp_offload_active (netdev, queue)) +nvme_tcp_offload_apply_limits (queue, netdev); +/* +* release the device as no offload context is +* established yet. +*/ +dev_put (netdev);
Quoted 1 line
the put is unclear, what does it pair with? the
get_netdev_for_sock?
Yes, get_netdev_for_sock() takes a reference, which we don't need at that point so we put it.
Sagi Grimberg
maintainerRe: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 11 lines
Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 344
Quoted 6 lines
drivers/nvme/host/tcp.c +5
-0+static inline bool is_netdev_ulp_offload_active (struct net_device *netdev, +struct nvme_tcp_queue *queue) +{ +if (!netdev || !queue) +return false;Is it reasonable to be called here with !netdev or !queue ?
The check is needed only for the IO queue case but we can move it earlier in
nvme_tcp_start_queue().
I still don't understand even on io queues how this can happen.
Quoted 14 lines
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 352
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ +/* If we cannot query the netdev limitations, do not offload */ +if (!nvme_tcp_ddp_query_limits (netdev, queue)) +return false; + +/* If netdev supports nvme-tcp ddp offload, we can offload */ +if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active)) +return true;This should be coming from the API itself, have the limits query api fail if this is off.
We can move the function to the ULP DDP layer.
Quoted 7 lines
Quoted 2 lines
btw, what is the active thing? is this driven from ethtool enable? what happens if the user disables it while there is a ulp using it?
The active bits are indeed driven by ethtool according to the design Jakub suggested.
The nvme-tcp connection will have to be reconnected to see the effect of changing the bit.
It should move inside the api as well. Don't want to care about it in nvme.
Quoted 7 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + return false;This can be folded to the above function.
We won't be able to check for TLS in a common wrapper. We think this should be kept.
Why? any tcp ddp need to be able to support tls. Nothing specific to nvme here.
Quoted 11 lines
Quoted 10 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 368
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +int ret; + + config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ;Question, what happens if the pfv changes, is the ddp guaranteed to work?
Quoted 3 lines
The existing HW supports only
NVME_TCP_PFV_1_0.
Once a new version will be used, the device driver should fail thesk_add().
OK.
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
drivers/nvme/host/tcp.c +37
-0+/* In presence of packet drops or network packet reordering, the device may lose +* synchronization between the TCP stream and the L5P framing, and require a +* resync with the kernel's TCP stack. +* +* - NIC HW identifies a PDU header at some TCP sequence number, +* and asks NVMe-TCP to confirm it. +* - When NVMe-TCP observes the requested TCP sequence, it will compare +* it with the PDU header TCP sequence, and report the result to the +* NIC driver +*/ +static void nvme_tcp_resync_response (struct nvme_tcp_queue *queue, +struct sk_buff *skb,unsigned int offset) +{ +u64 pdu_seq =TCP_SKB_CB (skb)->seq + offset - queue->pdu_offset; +struct net_device *netdev = queue->ctrl->offloading_netdev; +u64 pdu_val = (pdu_seq <<32 ) |ULP_DDP_RESYNC_PENDING ; +u64 resync_val; +u32 resync_seq; + + resync_val =atomic64_read (&queue->resync_req); +/* Lower 32 bit flags. Check validity of the request */ +if ((resync_val &ULP_DDP_RESYNC_PENDING ) ==0 ) +return ; + +/* +* Obtain and check requested sequence number: is this PDU header +* before the request? +*/ + resync_seq = resync_val >>32 ; +if (before (pdu_seq, resync_seq)) +return ; + +/* +* The atomic operation guarantees that we don't miss any NIC driver +* resync requests submitted after the above checks. +*/ + if (atomic64_cmpxchg(&queue->resync_req, pdu_val,Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 468
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ pdu_val & ~ULP_DDP_RESYNC_PENDING) != + atomic64_read(&queue->resync_req)) + netdev->netdev_ops->ulp_ddp_ops->resync(netdev, + queue->sock->sk, + pdu_seq);Who else is doing an atomic on this value? and what happens if the cmpxchg fails?
Quoted 7 lines
The driver thread can set
queue->resync_reqconcurrently in patch "net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload" in functionnvmeotcp_update_resync().If the cmpxchg fails it means a new resync request was triggered by the HW, the old request will be dropped and the new one will be processed by a later PDU.
So resync_req is actually the current tcp sequence number or something? The name resync_req is very confusing.
Quoted 39 lines
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+} + +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags) +{ +struct nvme_tcp_queue *queue = sk->sk_user_data; + +/* +* "seq" (TCP seq number) is what the HW assumes is the +* beginning of a PDU. The nvme-tcp layer needs to store the Quoted 8 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 481
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+* number along with the "flags" (ULP_DDP_RESYNC_PENDING) to +* indicate that a request is pending. +*/ +atomic64_set (&queue->resync_req, (((uint64_t )seq <<32 ) | flags));Question, is this coming from multiple contexts? what contexts are competing here that make it an atomic operation? It is unclear what is going on here tbh.
The driver could get a resync request and set
queue->resync_reqconcurrently while processing HW CQEs as you can see in patch "net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload" in functionnvmeotcp_update_resync().The resync flow is:
nvme-tcp mlx5 hw
| | |
| | sends CQE with
| | resync request
| | <----------------------'
|nvmeotcp_update_resync()
nvme_tcp_resync_request()<-----------'|
we store the requestLater, while receiving PDUs we check for pending requests. If there is one, we send call
nvme_tcp_resync_response()which calls into mlx5 to send the response to the HW.
...
Quoted 24 lines
Quoted 21 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+ ret =nvme_tcp_offload_socket (queue); +if (ret) { +dev_info (nctrl->device, +"failed to setup offload on queue %d ret=%d\n" , + idx, ret); + } + } + }else { ret =nvmf_connect_admin_queue (nctrl); +if (ret)drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2084
Quoted 10 lines
drivers/nvme/host/tcp.c +2
-7+ goto err; -if (!ret) { -set_bit (NVME_TCP_Q_LIVE , &queue->flags); - }else { -if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) -__nvme_tcp_stop_queue (queue); -dev_err (nctrl->device, -"failed to connect queue: %d ret=%d\n" , idx, ret); + netdev =get_netdev_for_sock (queue->sock->sk);Quoted 2 lines
Is there any chance that this is a different netdev than what is already recorded? doesn't make sense to me.
Quoted 6 lines
The idea is that we are first starting the admin queue, which looks up the netdev associated with the socket and stored in the queue. Later, when the IO queues are started, we use the recorded netdev.
In cases of bonding or vlan, a netdev can have lower device links, which
get_netdev_for_sock()will look up.
I think the code should in high level do:
if (idx) {
ret = nvmf_connect_io_queue(nctrl, idx);
if (ret)
goto err;
if (nvme_tcp_ddp_query_limits(queue))
nvme_tcp_offload_socket(queue);
} else {
ret = nvmf_connect_admin_queue(nctrl);
if (ret)
goto err;
ctrl->ddp_netdev = get_netdev_for_sock(queue->sock->sk);
if (nvme_tcp_ddp_query_limits(queue))
nvme_tcp_offload_apply_limits(queue);
}ctrl->ddp_netdev should be cleared and put when the admin queue is stopped/freed, similar to how async_req is handled.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ goto done; + } +if (is_netdev_ulp_offload_active (netdev, queue)) +nvme_tcp_offload_apply_limits (queue, netdev); +/* +* release the device as no offload context is +* established yet. +*/ +dev_put (netdev);
Quoted 4 lines
Quoted 1 line
the put is unclear, what does it pair with? the
get_netdev_for_sock?Yes,
get_netdev_for_sock()takes a reference, which we don't need at that point so we put it.
Well, you store a pointer to it, what happens if it goes away while the controller is being set up?
Re: [PATCH v12 07/26] nvme-tcp: Add DDP offload control path
Quoted 9 lines · Sagi Grimberg
Quoted 7 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!netdev || !queue) + return false;Is it reasonable to be called here with !netdev or !queue ?
The check is needed only for the IO queue case but we can move it earlier in
nvme_tcp_start_queue().I still don't understand even on io queues how this can happen.
In case where the netdev does not support DDP, when the admin queue is started, netdev->offloading_netdev will not be set and therefore will be NULL.
Later, when the IO queue is started:
netdev = queue->ctrl->offloading_netdev; <== NULL
if (is_netdev_ulp_offload_active(netdev, queue)) { <== we pass NULLWe can move the NULL check higher-up if you prefer, like so:
if (queue->ctrl->offloading_netdev &&
is_netdev_ulp_offload_active(queue->ctrl->offloading_netdev, queue)) {Quoted 17 lines
Quoted 17 lines
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 352
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+ +/* If we cannot query the netdev limitations, do not offload */ +if (!nvme_tcp_ddp_query_limits (netdev, queue)) +return false; + +/* If netdev supports nvme-tcp ddp offload, we can offload */ +if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active)) +return true;This should be coming from the API itself, have the limits query api fail if this is off.
We can move the function to the ULP DDP layer.
Quoted 2 lines
btw, what is the active thing? is this driven from ethtool enable? what happens if the user disables it while there is a ulp using it?
Quoted 7 lines
Quoted 4 lines
The active bits are indeed driven by ethtool according to the design Jakub suggested.
The nvme-tcp connection will have to be reconnected to see the effect of changing the bit.It should move inside the api as well. Don't want to care about it in nvme.
Ok, we will move it there.
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + return false;This can be folded to the above function.
We won't be able to check for TLS in a common wrapper. We think this should be kept.
Why? any tcp ddp need to be able to support tls. Nothing specific to nvme here.
True, we will move it to the ULP wrapper.
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
drivers/nvme/host/tcp.c static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, · line 468
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+/* +* The atomic operation guarantees that we don't miss any NIC driver +* resync requests submitted after the above checks. +*/ +if (atomic64_cmpxchg (&queue->resync_req, pdu_val, + pdu_val & ~ULP_DDP_RESYNC_PENDING ) != +atomic64_read (&queue->resync_req)) + netdev->netdev_ops->ulp_ddp_ops->resync (netdev, + queue->sock->sk, + pdu_seq);Quoted 2 lines
Who else is doing an atomic on this value? and what happens if the cmpxchg fails?
The driver thread can set
queue->resync_reqconcurrently in patch "net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload" in functionnvmeotcp_update_resync().
Quoted 6 lines
Quoted 3 lines
If the cmpxchg fails it means a new resync request was triggered by the HW, the old request will be dropped and the new one will be processed by a later PDU.
So
resync_reqis actually the current tcp sequence number or something? The nameresync_reqis very confusing.
queue->resync_req is the TCP sequence for which the HW requested a resync operation. We can rename it with queue->resync_tcp_seq.
Quoted 32 lines
Quoted 24 lines
Quoted 21 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+ ret =nvme_tcp_offload_socket (queue); +if (ret) { +dev_info (nctrl->device, +"failed to setup offload on queue %d ret=%d\n" , + idx, ret); + } + } + }else { ret =nvmf_connect_admin_queue (nctrl); +if (ret)drivers/nvme/host/tcp.c static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) · line 2084
Quoted 10 lines
drivers/nvme/host/tcp.c +2
-7+ goto err; -if (!ret) { -set_bit (NVME_TCP_Q_LIVE , &queue->flags); - }else { -if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) -__nvme_tcp_stop_queue (queue); -dev_err (nctrl->device, -"failed to connect queue: %d ret=%d\n" , idx, ret); + netdev =get_netdev_for_sock (queue->sock->sk);Quoted 2 lines
Is there any chance that this is a different netdev than what is already recorded? doesn't make sense to me.
Quoted 6 lines
The idea is that we are first starting the admin queue, which looks up the netdev associated with the socket and stored in the queue. Later, when the IO queues are started, we use the recorded netdev.
In cases of bonding or vlan, a netdev can have lower device links, which
get_netdev_for_sock()will look up.I think the code should in high level do:
Quoted 15 lines
if (idx) {
ret = nvmf_connect_io_queue(nctrl, idx);
if (ret)
goto err;
if (nvme_tcp_ddp_query_limits(queue))
nvme_tcp_offload_socket(queue);
} else {
ret = nvmf_connect_admin_queue(nctrl);
if (ret)
goto err;
ctrl->ddp_netdev = get_netdev_for_sock(queue->sock->sk);
if (nvme_tcp_ddp_query_limits(queue))
nvme_tcp_offload_apply_limits(queue);
}Ok, we will follow this design.
Quoted 2 lines
ctrl->ddp_netdevshould be cleared and put when the admin queue is stopped/freed, similar to howasync_reqis handled.
Thanks, we will clear ddp_netdev on queue stop/free.
This will also prevent reusing a potentially wrong netdev after a reconnection.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+/* +* release the device as no offload context is +* established yet. +*/ +dev_put (netdev);
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
the put is unclear, what does it pair with? the
get_netdev_for_sock?Yes,
get_netdev_for_sock()takes a reference, which we don't need at that point so we put it.Well, you store a pointer to it, what happens if it goes away while the controller is being set up?
It's a problem. We will remove the dev_put() to keep the first reference, and only release it when it is cleared from the admin queue.
PATCH v12 8/26 nvme-tcp: Add DDP data-path
In reply to Aurelien AptelFrom: Boris Pismenny <[email protected]>
Introduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id in the PDU.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.
The data-path interface contains two routines: setup/teardown. The setup provides the mapping from command_id to the request buffers, while the teardown removes this mapping.
For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the generic mq_blk completion.
--- drivers/nvme/host/tcp.c | 121 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 116 insertions(+), 5 deletions(-)
drivers/nvme/host/tcp.c +116 -5
@@ -116,6 +116,13 @@ struct nvme_tcp_request {size_t offset;size_t data_sent;enum nvme_tcp_send_state state; + +#ifdef CONFIG_ULP_DDP +bool offloaded; +struct ulp_ddp_io ddp; +__le16 ddp_status; +union nvme_result result; +#endif };enum nvme_tcp_queue_flags {
@@ -354,11 +361,75 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static int nvme_tcp_req_map_ddp_sg (struct nvme_tcp_request *req,struct request *rq) +{ +int ret; + + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret =sg_alloc_table_chained (&req->ddp.sg_table, +blk_rq_nr_phys_segments (rq), + req->ddp.sg_table.sgl,SG_CHUNK_SIZE ); +if (ret) +return -ENOMEM ; + req->ddp.nents =blk_rq_map_sg (rq->q, rq, req->ddp.sg_table.sgl); +return 0 ; +} +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +static void nvme_tcp_ddp_teardown_done (void *ddp_ctx);static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { .resync_request = nvme_tcp_resync_request, + .ddp_teardown_done = nvme_tcp_ddp_teardown_done, }; +static void nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +struct request *rq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + + netdev->netdev_ops->ulp_ddp_ops->teardown (netdev, queue->sock->sk, + &req->ddp, rq); +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); +} + +static void nvme_tcp_ddp_teardown_done (void *ddp_ctx) +{ +struct request *rq = ddp_ctx; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (!nvme_try_complete_req (rq, req->ddp_status, req->result)) +nvme_complete_rq (rq); +} + +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id, +struct request *rq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +int ret; + +if (rq_data_dir (rq) !=READ || + queue->ctrl->offload_io_threshold >blk_rq_payload_bytes (rq)) +return 0 ; + + req->ddp.command_id = command_id; + ret =nvme_tcp_req_map_ddp_sg (req, rq); +if (ret) +return -ENOMEM ; + + ret = netdev->netdev_ops->ulp_ddp_ops->setup (netdev, queue->sock->sk, + &req->ddp); +if (ret) { +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); +return ret; + } + +/* if successful, sg table is freed in nvme_tcp_teardown_ddp() */ + req->offloaded = true; +return 0 ; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) {struct net_device *netdev = queue->ctrl->offloading_netdev;
@@ -491,6 +562,12 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id, +struct request *rq) +{ +return 0 ; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) {return -EOPNOTSUPP ;
@@ -778,6 +855,26 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)queue_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work); } +static void nvme_tcp_complete_request (struct request *rq, +__le16 status, +union nvme_result result, +__u16 command_id) +{ +#ifdef CONFIG_ULP_DDP +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (req->offloaded) { + req->ddp_status = status; + req->result = result; +nvme_tcp_teardown_ddp (req->queue, rq); +return ; + } +#endif + +if (!nvme_try_complete_req (rq, status, result)) +nvme_complete_rq (rq); +} +static int nvme_tcp_process_nvme_cqe (struct nvme_tcp_queue *queue,struct nvme_completion *cqe) {
@@ -797,10 +894,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,if (req->status ==cpu_to_le16 (NVME_SC_SUCCESS )) req->status = cqe->status; -if (!nvme_try_complete_req (rq, req->status, cqe->result)) -nvme_complete_rq (rq); +nvme_tcp_complete_request (rq, req->status, cqe->result, + cqe->command_id); queue->nr_cqe++; -return 0 ; }
@@ -998,10 +1094,13 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,static inline void nvme_tcp_end_request (struct request *rq,u16 status) { +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_tcp_queue *queue = req->queue; +struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;union nvme_result res = {}; -if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) -nvme_complete_rq (rq); +nvme_tcp_complete_request (rq,cpu_to_le16 (status <<1 ), res, + pdu->command_id); } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -1308,6 +1407,15 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else msg.msg_flags |=MSG_EOR ; +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) { + ret =nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, +blk_mq_rq_from_pdu (req)); +WARN_ONCE (ret,"ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)" , +nvme_tcp_queue_id (queue), + pdu->cmd.common.command_id, + ret); + } +if (queue->hdr_digest && !req->offset)nvme_tcp_hdgst (queue->snd_hash, pdu,sizeof (*pdu));
@@ -2739,6 +2847,9 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,if (ret)return ret; +#ifdef CONFIG_ULP_DDP + req->offloaded = false; +#endif req->state =NVME_TCP_SEND_CMD_PDU ; req->status =cpu_to_le16 (NVME_SC_SUCCESS ); req->offset =0 ;
Signature · 1 line
2.34.1
Sagi Grimberg
maintainerRe: [PATCH v12 08/26] nvme-tcp: Add DDP data-path
Quoted 47 lines · Aurelien Aptel
From: Boris Pismenny <[email protected]>
Introduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to thecommand_idin the PDU.
To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages.The data-path interface contains two routines: setup/teardown. The setup provides the mapping from
command_idto the request buffers, while the teardown removes this mapping.For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the genericmq_blkcompletion.Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>--- drivers/nvme/host/tcp.c | 121 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 116 insertions(+), 5 deletions(-)drivers/nvme/host/tcp.c +7
-0@@ -116,6 +116,13 @@ struct nvme_tcp_request {size_t offset;size_t data_sent;enum nvme_tcp_send_state state; + +#ifdef CONFIG_ULP_DDP +bool offloaded; +struct ulp_ddp_io ddp; +__le16 ddp_status; +union nvme_result result; +#endif };enum nvme_tcp_queue_flags {
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 364
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0@@ -354,11 +361,75 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static int nvme_tcp_req_map_ddp_sg (struct nvme_tcp_request *req,struct request *rq)
Why do you pass both req and rq? You can derive each from the other.
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 374
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+{ +int ret; + + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret =sg_alloc_table_chained (&req->ddp.sg_table, +blk_rq_nr_phys_segments (rq), + req->ddp.sg_table.sgl,SG_CHUNK_SIZE ); +if (ret) +return -ENOMEM ; + req->ddp.nents =blk_rq_map_sg (rq->q, rq, req->ddp.sg_table.sgl);
General question, I'm assuming that the hca knows how to deal with a controller that sends c2hdata in parts?
Quoted 22 lines
drivers/nvme/host/tcp.c +17
-0+return 0 ; +} +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags); +static void nvme_tcp_ddp_teardown_done (void *ddp_ctx);static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { .resync_request = nvme_tcp_resync_request, + .ddp_teardown_done = nvme_tcp_ddp_teardown_done, }; +static void nvme_tcp_teardown_ddp (struct nvme_tcp_queue *queue, +struct request *rq) +{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + + netdev->netdev_ops->ulp_ddp_ops->teardown (netdev, queue->sock->sk, + &req->ddp, rq); +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); +} + +static void nvme_tcp_ddp_teardown_done (void *ddp_ctx)
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 406
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+{ +struct request *rq = ddp_ctx; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (!nvme_try_complete_req (rq, req->ddp_status, req->result)) +nvme_complete_rq (rq); +} + +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id, +struct request *rq)
I think you can use nvme_cid(rq) instead of passing the command_id.
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 417
Quoted 11 lines
drivers/nvme/host/tcp.c +11
-0+{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +int ret; + +if (rq_data_dir (rq) !=READ || + queue->ctrl->offload_io_threshold >blk_rq_payload_bytes (rq)) +return 0 ; + + req->ddp.command_id = command_id; + ret =nvme_tcp_req_map_ddp_sg (req, rq);
Don't see why map_ddp_sg is not open-coded here, its the only call-site, and its pretty much does exactly what its called.
Quoted 89 lines
drivers/nvme/host/tcp.c +48
-5+if (ret) +return -ENOMEM ; + + ret = netdev->netdev_ops->ulp_ddp_ops->setup (netdev, queue->sock->sk, + &req->ddp); +if (ret) { +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); +return ret; + } + +/* if successful, sg table is freed in nvme_tcp_teardown_ddp() */ + req->offloaded = true; +return 0 ; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) {struct net_device *netdev = queue->ctrl->offloading_netdev;@@ -491,6 +562,12 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id, +struct request *rq) +{ +return 0 ; +} +static int nvme_tcp_offload_socket (struct nvme_tcp_queue *queue) {return -EOPNOTSUPP ;@@ -778,6 +855,26 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)queue_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work); } +static void nvme_tcp_complete_request (struct request *rq, +__le16 status, +union nvme_result result, +__u16 command_id) +{ +#ifdef CONFIG_ULP_DDP +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); + +if (req->offloaded) { + req->ddp_status = status; + req->result = result; +nvme_tcp_teardown_ddp (req->queue, rq); +return ; + } +#endif + +if (!nvme_try_complete_req (rq, status, result)) +nvme_complete_rq (rq); +} +static int nvme_tcp_process_nvme_cqe (struct nvme_tcp_queue *queue,struct nvme_completion *cqe) {@@ -797,10 +894,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,if (req->status ==cpu_to_le16 (NVME_SC_SUCCESS )) req->status = cqe->status; -if (!nvme_try_complete_req (rq, req->status, cqe->result)) -nvme_complete_rq (rq); +nvme_tcp_complete_request (rq, req->status, cqe->result, + cqe->command_id); queue->nr_cqe++; -return 0 ; }@@ -998,10 +1094,13 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,static inline void nvme_tcp_end_request (struct request *rq,u16 status) { +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_tcp_queue *queue = req->queue; +struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;union nvme_result res = {}; -if (!nvme_try_complete_req (rq,cpu_to_le16 (status <<1 ), res)) -nvme_complete_rq (rq); +nvme_tcp_complete_request (rq,cpu_to_le16 (status <<1 ), res, + pdu->command_id); } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,@@ -1308,6 +1407,15 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1417
Quoted 10 lines
drivers/nvme/host/tcp.c +8
-0msg.msg_flags |=MSG_EOR ; +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) { + ret =nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, +blk_mq_rq_from_pdu (req)); +WARN_ONCE (ret,"ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)" , +nvme_tcp_queue_id (queue), + pdu->cmd.common.command_id, + ret); + }
Any reason why this is done here when sending the command pdu and not in setup time?
Quoted 14 lines
drivers/nvme/host/tcp.c +4
-0+if (queue->hdr_digest && !req->offset)nvme_tcp_hdgst (queue->snd_hash, pdu,sizeof (*pdu));@@ -2739,6 +2847,9 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,if (ret)return ret; +#ifdef CONFIG_ULP_DDP + req->offloaded = false; +#endif req->state =NVME_TCP_SEND_CMD_PDU ; req->status =cpu_to_le16 (NVME_SC_SUCCESS ); req->offset =0 ;
Re: [PATCH v12 08/26] nvme-tcp: Add DDP data-path
Quoted 4 lines · Sagi Grimberg
Quoted 1 line · Aurelien Aptel
+static int
nvme_tcp_req_map_ddp_sg(struct nvme_tcp_request *req, struct request *rq)Why do you pass both req and rq? You can derive each from the other.
Thanks, we will remove the redundant parameter.
Quoted 5 lines
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+{ +int ret; + + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret = sg_alloc_table_chained(&req->ddp.sg_table,
Quoted 8 lines
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 374
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+blk_rq_nr_phys_segments (rq), + req->ddp.sg_table.sgl, SG_CHUNK_SIZE); +if (ret) +return -ENOMEM ; + req->ddp.nents =blk_rq_map_sg (rq->q, rq, req->ddp.sg_table.sgl);General question, I'm assuming that the hca knows how to deal with a controller that sends c2hdata in parts?
Yes, the hardware supports the offloading of multiple c2hdata PDUs per IO.
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+static int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue, u16 command_id, + struct request *rq)I think you can use
nvme_cid(rq)instead of passing thecommand_id.
Thanks, we will use it.
Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+{ +struct net_device *netdev = queue->ctrl->offloading_netdev; +struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +int ret; + + if (rq_data_dir(rq) != READ ||
Quoted 8 lines
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 417
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ queue->ctrl->offload_io_threshold > blk_rq_payload_bytes(rq)) +return 0 ; + + req->ddp.command_id = command_id; + ret =nvme_tcp_req_map_ddp_sg (req, rq);Don't see why
map_ddp_sgis not open-coded here, its the only call-site, and its pretty much does exactly what its called.
Sure, we will open code it.
Quoted 4 lines
Quoted 4 lines
drivers/nvme/host/tcp.c +0
-0@@ -1308,6 +1407,15 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else msg.msg_flags |=MSG_EOR ;
Quoted 11 lines
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1417
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) { + ret =nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, +blk_mq_rq_from_pdu (req)); +WARN_ONCE (ret,"ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)" , +nvme_tcp_queue_id (queue), + pdu->cmd.common.command_id, + ret); + }Any reason why this is done here when sending the command pdu and not in setup time?
We wish to interact with the HW from the same CPU per queue, hence we are calling setup_ddp() after queue->io_cpu == raw_smp_processor_id() was checked in nvme_tcp_queue_request().
Sagi Grimberg
maintainerRe: [PATCH v12 08/26] nvme-tcp: Add DDP data-path
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
drivers/nvme/host/tcp.c +0
-0@@ -1308,6 +1407,15 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else msg.msg_flags |=MSG_EOR ;Quoted 11 lines
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1417
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) { + ret =nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, +blk_mq_rq_from_pdu (req)); +WARN_ONCE (ret,"ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)" , +nvme_tcp_queue_id (queue), + pdu->cmd.common.command_id, + ret); + }Any reason why this is done here when sending the command pdu and not in setup time?
Quoted 3 lines
We wish to interact with the HW from the same CPU per queue, hence we are calling
setup_ddp()afterqueue->io_cpu==raw_smp_processor_id()was checked innvme_tcp_queue_request().
That is very fragile. You cannot depend on this micro-optimization being in the code. Is this related to a hidden steering rule you are adding to the hw?
Which reminds me, in the control patch, you are passing io_cpu, this is
also a dependency that should be avoided, you should use the same mechanism as arfs to learn where the socket is being reaped.
Re: [PATCH v12 08/26] nvme-tcp: Add DDP data-path
Quoted 16 lines · Sagi Grimberg
Quoted 16 lines
Quoted 15 lines
drivers/nvme/host/tcp.c static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) · line 1417
Quoted 12 lines
drivers/nvme/host/tcp.c +8
-0@@ -1308,6 +1407,15 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)else msg.msg_flags |=MSG_EOR ; +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) { + ret =nvme_tcp_setup_ddp (queue, pdu->cmd.common.command_id, +blk_mq_rq_from_pdu (req)); +WARN_ONCE (ret,"ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)" , +nvme_tcp_queue_id (queue), + pdu->cmd.common.command_id, + ret); + }Any reason why this is done here when sending the command pdu and not in setup time?
Quoted 7 lines
Quoted 3 lines
We wish to interact with the HW from the same CPU per queue, hence we are calling
setup_ddp()afterqueue->io_cpu==raw_smp_processor_id()was checked innvme_tcp_queue_request().That is very fragile. You cannot depend on this micro-optimization being in the code. Is this related to a hidden steering rule you are adding to the hw?
We are using a steering rule in order to redirect packets into the offload engine. This rule also helps with aligning the nvme-tcp connection with a specific core.
Quoted 3 lines
Which reminds me, in the control patch, you are passing
io_cpu, this is also a dependency that should be avoided, you should use the same mechanism as arfs to learn where the socket is being reaped.
We can use queue->sock->sk->sk_incoming_cpu instead of queue->io_cpu as it is used in the nvme-tcp target.
PATCH v12 9/26 nvme-tcp: RX DDGST offload
In reply to Aurelien AptelFrom: Yoray Zack <[email protected]>
Enable rx side of DDGST offload when supported.
At the end of the capsule, check if all the skb bits are on, and if not recalculate the DDGST in SW and check it.
--- drivers/nvme/host/tcp.c | 120 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 112 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +112 -8
@@ -130,6 +130,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 ,NVME_TCP_Q_OFF_DDP =3 , + NVME_TCP_Q_OFF_DDGST_RX = 4, }; enum nvme_tcp_recv_state {
@@ -168,6 +169,7 @@ struct nvme_tcp_queue {*/ atomic64_t resync_req;struct ulp_ddp_limits ddp_limits; +bool ddp_ddgst_valid;#endif /* send state */
@@ -358,9 +360,29 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active))return true; +/* +* Otherwise, if netdev supports nvme-tcp ddgst offload and it +* was enabled, we can offload +*/ +if (queue->data_digest &&test_bit (ULP_DDP_C_NVME_TCP_DDGST_RX_BIT , + netdev->ulp_ddp_caps.active)) +return true; +return false; } +static inline bool nvme_tcp_ddp_ddgst_ok (struct nvme_tcp_queue *queue) +{ +return queue->ddp_ddgst_valid; +} + +static inline void nvme_tcp_ddp_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb) +{ +if (queue->ddp_ddgst_valid) + queue->ddp_ddgst_valid =skb_is_ulp_crc (skb); +} +static int nvme_tcp_req_map_ddp_sg (struct nvme_tcp_request *req,struct request *rq) {int ret;
@@ -375,6 +397,38 @@ static int nvme_tcp_req_map_ddp_sg(struct nvme_tcp_request *req, struct requestreturn 0 ; } +static void nvme_tcp_ddp_ddgst_recalc (struct ahash_request *hash, +struct request *rq, +__le32 *ddgst) +{ +struct nvme_tcp_request *req; + +if (!rq) +return ; + + req =blk_mq_rq_to_pdu (rq); + +if (!req->offloaded) { +/* if we have DDGST_RX offload without DDP the request +* wasn't mapped, so we need to map it here +*/ +if (nvme_tcp_req_map_ddp_sg (req, rq)) +return ; + } + + req->ddp.sg_table.sgl = req->ddp.first_sgl; +ahash_request_set_crypt (hash, req->ddp.sg_table.sgl, (u8 *)ddgst, + req->data_len); +crypto_ahash_digest (hash); + +if (!req->offloaded) { +/* without DDP, ddp_teardown() won't be called, so +* free the table here +*/ +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); + } +} +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags);static void nvme_tcp_ddp_teardown_done (void *ddp_ctx);static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
@@ -434,6 +488,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) {struct net_device *netdev = queue->ctrl->offloading_netdev;struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +bool offload_ddp =test_bit (ULP_DDP_C_NVME_TCP_BIT , + netdev->ulp_ddp_caps.active); +bool offload_ddgst_rx =test_bit (ULP_DDP_C_NVME_TCP_DDGST_RX_BIT , + netdev->ulp_ddp_caps.active);int ret; config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ;
@@ -460,7 +518,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) }inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; -set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +if (offload_ddp) +set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +if (queue->data_digest && offload_ddgst_rx) +set_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags);return 0 ; }
@@ -474,6 +535,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) }clear_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +clear_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags); netdev->netdev_ops->ulp_ddp_ops->sk_del (netdev, queue->sock->sk);
@@ -562,6 +624,20 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static inline bool nvme_tcp_ddp_ddgst_ok (struct nvme_tcp_queue *queue) +{ +return false; +} + +static inline void nvme_tcp_ddp_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb) +{} + +static void nvme_tcp_ddp_ddgst_recalc (struct ahash_request *hash, +struct request *rq, +__le32 *ddgst) +{} +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id,struct request *rq) {
@@ -844,6 +920,9 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->pdu_offset =0 ; queue->data_remaining =-1 ; queue->ddgst_remaining =0 ; +#ifdef CONFIG_ULP_DDP + queue->ddp_ddgst_valid = true; +#endif }static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl)
@@ -1047,7 +1126,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))nvme_tcp_resync_response (queue, skb, *offset); ret = skb_copy_bits(skb, *offset,
@@ -1111,6 +1191,10 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,nvme_cid_to_rq (nvme_tcp_tagset (queue), pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +if (queue->data_digest && +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) +nvme_tcp_ddp_ddgst_update (queue, skb); +while (true) {int recv_len, ret;
@@ -1139,7 +1223,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len =min_t (size_t, recv_len,iov_iter_count (&req->iter)); -if (queue->data_digest) +if (queue->data_digest && + !test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) ret =skb_copy_and_hash_datagram_iter (skb, *offset, &req->iter, recv_len, queue->rcv_hash);else
@@ -1181,8 +1266,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; +struct request *rq;int ret; +if (test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) +nvme_tcp_ddp_ddgst_update (queue, skb); ret =skb_copy_bits (skb, *offset, &ddgst[off], recv_len);if (unlikely (ret))return ret;
@@ -1193,9 +1281,25 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,if (queue->ddgst_remaining)return 0 ; + rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), + pdu->command_id); + +if (test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) { +/* +* If HW successfully offloaded the digest +* verification, we can skip it +*/ +if (nvme_tcp_ddp_ddgst_ok (queue)) + goto out; +/* +* Otherwise we have to recalculate and verify the +* digest with the software-fallback +*/ +nvme_tcp_ddp_ddgst_recalc (queue->rcv_hash, rq, + &queue->exp_ddgst); + } +if (queue->recv_ddgst != queue->exp_ddgst) { -struct request *rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), - pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); req->status =cpu_to_le16 (NVME_SC_DATA_XFER_ERROR );
@@ -1206,9 +1310,8 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, le32_to_cpu(queue->exp_ddgst)); } +out:if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) { -struct request *rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), - pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq);nvme_tcp_end_request (rq,le16_to_cpu (req->status));
@@ -2125,7 +2228,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_ops (queue);cancel_work_sync (&queue->io_work); -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))nvme_tcp_unoffload_socket (queue); }
Signature · 1 line
2.34.1
Sagi Grimberg
maintainerRe: [PATCH v12 09/26] nvme-tcp: RX DDGST offload
Quoted 79 lines · Aurelien Aptel
From: Yoray Zack <[email protected]>
Enable rx side of DDGST offload when supported.
At the end of the capsule, check if all the skb bits are on, and if not recalculate the DDGST in SW and check it.
Signed-off-by Yoray Zack <[email protected]>Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>--- drivers/nvme/host/tcp.c | 120 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 112 insertions(+), 8 deletions(-)drivers/nvme/host/tcp.c +29
-0@@ -130,6 +130,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 ,NVME_TCP_Q_OFF_DDP =3 , + NVME_TCP_Q_OFF_DDGST_RX = 4, }; enum nvme_tcp_recv_state {@@ -168,6 +169,7 @@ struct nvme_tcp_queue {*/ atomic64_t resync_req;struct ulp_ddp_limits ddp_limits; +bool ddp_ddgst_valid;#endif /* send state */ @@ -358,9 +360,29 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active))return true; +/* +* Otherwise, if netdev supports nvme-tcp ddgst offload and it +* was enabled, we can offload +*/ +if (queue->data_digest &&test_bit (ULP_DDP_C_NVME_TCP_DDGST_RX_BIT , + netdev->ulp_ddp_caps.active)) +return true; +return false; } +static inline bool nvme_tcp_ddp_ddgst_ok (struct nvme_tcp_queue *queue) +{ +return queue->ddp_ddgst_valid; +} + +static inline void nvme_tcp_ddp_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb) +{ +if (queue->ddp_ddgst_valid) + queue->ddp_ddgst_valid =skb_is_ulp_crc (skb); +} +static int nvme_tcp_req_map_ddp_sg (struct nvme_tcp_request *req,struct request *rq) {int ret;@@ -375,6 +397,38 @@ static int nvme_tcp_req_map_ddp_sg(struct nvme_tcp_request *req, struct requestreturn 0 ; } +static void nvme_tcp_ddp_ddgst_recalc (struct ahash_request *hash, +struct request *rq, +__le32 *ddgst) +{ +struct nvme_tcp_request *req; + +if (!rq)
drivers/nvme/host/tcp.c static int nvme_tcp_req_map_ddp_sg(struct nvme_tcp_request *req, struct request · line 416
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+return ; + + req =blk_mq_rq_to_pdu (rq); + +if (!req->offloaded) { +/* if we have DDGST_RX offload without DDP the request +* wasn't mapped, so we need to map it here +*/ +if (nvme_tcp_req_map_ddp_sg (req, rq)) +return ;
grr.. wondering if this is something we want to support (crc without ddp).
drivers/nvme/host/tcp.c static int nvme_tcp_req_map_ddp_sg(struct nvme_tcp_request *req, struct request · line 419
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ } + + req->ddp.sg_table.sgl = req->ddp.first_sgl;
Why is this assignment needed? why not pass req->ddp.first_sgl ?
Quoted 78 lines
drivers/nvme/host/tcp.c +38
-1+ahash_request_set_crypt (hash, req->ddp.sg_table.sgl, (u8 *)ddgst, + req->data_len); +crypto_ahash_digest (hash); + +if (!req->offloaded) { +/* without DDP, ddp_teardown() won't be called, so +* free the table here +*/ +sg_free_table_chained (&req->ddp.sg_table,SG_CHUNK_SIZE ); + } +} +static bool nvme_tcp_resync_request (struct sock *sk,u32 seq,u32 flags);static void nvme_tcp_ddp_teardown_done (void *ddp_ctx);static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {@@ -434,6 +488,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) {struct net_device *netdev = queue->ctrl->offloading_netdev;struct ulp_ddp_config config = {.type =ULP_DDP_NVME }; +bool offload_ddp =test_bit (ULP_DDP_C_NVME_TCP_BIT , + netdev->ulp_ddp_caps.active); +bool offload_ddgst_rx =test_bit (ULP_DDP_C_NVME_TCP_DDGST_RX_BIT , + netdev->ulp_ddp_caps.active);int ret; config.nvmeotcp.pfv =NVME_TCP_PFV_1_0 ;@@ -460,7 +518,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) }inet_csk (queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; -set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +if (offload_ddp) +set_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +if (queue->data_digest && offload_ddgst_rx) +set_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags);return 0 ; }@@ -474,6 +535,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) }clear_bit (NVME_TCP_Q_OFF_DDP , &queue->flags); +clear_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags); netdev->netdev_ops->ulp_ddp_ops->sk_del (netdev, queue->sock->sk);@@ -562,6 +624,20 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,return false; } +static inline bool nvme_tcp_ddp_ddgst_ok (struct nvme_tcp_queue *queue) +{ +return false; +} + +static inline void nvme_tcp_ddp_ddgst_update (struct nvme_tcp_queue *queue, +struct sk_buff *skb) +{} + +static void nvme_tcp_ddp_ddgst_recalc (struct ahash_request *hash, +struct request *rq, +__le32 *ddgst) +{} +static int nvme_tcp_setup_ddp (struct nvme_tcp_queue *queue,u16 command_id,struct request *rq) {@@ -844,6 +920,9 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->pdu_offset =0 ; queue->data_remaining =-1 ; queue->ddgst_remaining =0 ; +#ifdef CONFIG_ULP_DDP + queue->ddp_ddgst_valid = true; +#endif }static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl)
drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1130
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1@@ -1047,7 +1126,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))
This now becomes two atomic bitops to check for each capability, where its more likely that neighther are on...
Is this really racing with anything? maybe just check with bitwise AND? or a local variable (or struct member)
I don't think that we should add any more overhead for the normal path than we already have.
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1195
Quoted 9 lines
drivers/nvme/host/tcp.c +2
-0nvme_tcp_resync_response (queue, skb, *offset); ret = skb_copy_bits(skb, *offset,@@ -1111,6 +1191,10 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,nvme_cid_to_rq (nvme_tcp_tagset (queue), pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); +if (queue->data_digest && +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))
And a third atomic bitop..
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1227
Quoted 12 lines
drivers/nvme/host/tcp.c +4
-1+nvme_tcp_ddp_ddgst_update (queue, skb); +while (true) {int recv_len, ret;@@ -1139,7 +1223,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len =min_t (size_t, recv_len,iov_iter_count (&req->iter)); -if (queue->data_digest) +if (queue->data_digest && + !test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))
and a 4'th atomic bitop...
drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1273
Quoted 12 lines
drivers/nvme/host/tcp.c +3
-0ret =skb_copy_and_hash_datagram_iter (skb, *offset, &req->iter, recv_len, queue->rcv_hash); else@@ -1181,8 +1266,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; +struct request *rq;int ret; +if (test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) +nvme_tcp_ddp_ddgst_update (queue, skb);
and a 5'th atomic bitop...
drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1287
Quoted 11 lines
drivers/nvme/host/tcp.c +4
-0ret =skb_copy_bits (skb, *offset, &ddgst[off], recv_len);if (unlikely (ret))return ret;@@ -1193,9 +1281,25 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,if (queue->ddgst_remaining)return 0 ; + rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), + pdu->command_id); + +if (test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) {
and a 6'th... ok this is just spraying atomic bitops on the data path. Please find a better solution to this.
Quoted 41 lines
drivers/nvme/host/tcp.c +17
-5+/* +* If HW successfully offloaded the digest +* verification, we can skip it +*/ +if (nvme_tcp_ddp_ddgst_ok (queue)) + goto out; +/* +* Otherwise we have to recalculate and verify the +* digest with the software-fallback +*/ +nvme_tcp_ddp_ddgst_recalc (queue->rcv_hash, rq, + &queue->exp_ddgst); + } +if (queue->recv_ddgst != queue->exp_ddgst) { -struct request *rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), - pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq); req->status =cpu_to_le16 (NVME_SC_DATA_XFER_ERROR );@@ -1206,9 +1310,8 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, le32_to_cpu(queue->exp_ddgst)); } +out:if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) { -struct request *rq =nvme_cid_to_rq (nvme_tcp_tagset (queue), - pdu->command_id);struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq);nvme_tcp_end_request (rq,le16_to_cpu (req->status));@@ -2125,7 +2228,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)kernel_sock_shutdown (queue->sock,SHUT_RDWR );nvme_tcp_restore_sock_ops (queue);cancel_work_sync (&queue->io_work); -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))nvme_tcp_unoffload_socket (queue); }
Re: [PATCH v12 09/26] nvme-tcp: RX DDGST offload
Quoted 2 lines · Sagi Grimberg
grr.. wondering if this is something we want to support (crc without ddp).
We agree, we don't want to support it. We will remove it and check it doesn't happen in is_netdev_offload_active().
Quoted 2 lines
Quoted 1 line
+
req->ddp.sg_table.sgl=req->ddp.first_sgl;Why is this assignment needed? why not pass
req->ddp.first_sgl?
Correct, this assignment is not needed we will remove it.
Quoted 12 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1130
Quoted 8 lines
static void
nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)drivers/nvme/host/tcp.c +2
-1@@ -1047,7 +1126,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))This now becomes two atomic bitops to check for each capability, where its more likely that neighther are on...
Quoted 4 lines
Is this really racing with anything? maybe just check with bitwise AND? or a local variable (or struct member)
I don't think that we should add any more overhead for the normal path than we already have.
Are you sure test_bit() is atomic? The underlying definitions seems non-atomic (constant_test_bit or const_test_bit), are we missing anything?
We were also following a similar implementation to NVME_TCP_Q_POLLING which was using test_bit(). Should we move to a regular bool flag like queue->data_digest?
Quoted 4 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1195
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+if (queue->data_digest && +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))And a third atomic bitop..
See above
Quoted 2 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1227
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+ !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))and a 4'th atomic bitop...
See above
Quoted 4 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · line 1273
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+if (test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags)) +nvme_tcp_ddp_ddgst_update (queue, skb);and a 5'th atomic bitop...
See above
Quoted 3 lines
Quoted 1 line
+ if (
test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) {and a 6'th... ok this is just spraying atomic bitops on the data path. Please find a better solution to this.
See above
Thanks
Sagi Grimberg
maintainerRe: [PATCH v12 09/26] nvme-tcp: RX DDGST offload
Quoted 24 lines · Aurelien Aptel
Quoted 2 lines · Sagi Grimberg
grr.. wondering if this is something we want to support (crc without ddp).
We agree, we don't want to support it. We will remove it and check it doesn't happen in
is_netdev_offload_active().Quoted 2 lines
Quoted 1 line
+
req->ddp.sg_table.sgl=req->ddp.first_sgl;Why is this assignment needed? why not pass
req->ddp.first_sgl?Correct, this assignment is not needed we will remove it.
Quoted 12 lines
drivers/nvme/host/tcp.c static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, · line 1130
Quoted 8 lines
static void
nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)drivers/nvme/host/tcp.c +2
-1@@ -1047,7 +1126,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,size_t rcv_len =min_t (size_t, *len, queue->pdu_remaining);int ret; -if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags)) +if (test_bit (NVME_TCP_Q_OFF_DDP , &queue->flags) || +test_bit (NVME_TCP_Q_OFF_DDGST_RX , &queue->flags))This now becomes two atomic bitops to check for each capability, where its more likely that neighther are on...
Quoted 8 lines
Quoted 4 lines
Is this really racing with anything? maybe just check with bitwise AND? or a local variable (or struct member)
I don't think that we should add any more overhead for the normal path than we already have.Are you sure
test_bit()is atomic? The underlying definitions seems non-atomic (constant_test_bitorconst_test_bit), are we missing anything?
Hmm, no you're right.
This makes me wonder if we have some places to convert test_bit to test_bit_acquire where we must not make forward progress.
This particular condition is fine I think...
PATCH v12 10/26 nvme-tcp: Deal with netdevice DOWN events
In reply to Aurelien AptelFrom: Or Gerlitz <[email protected]>
For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.
These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroys them.
Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.
We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.
This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.
--- drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
drivers/nvme/host/tcp.c +39 -0
@@ -221,6 +221,7 @@ struct nvme_tcp_ctrl {static LIST_HEAD (nvme_tcp_ctrl_list);static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); +static struct notifier_block nvme_tcp_netdevice_nb;static struct workqueue_struct *nvme_tcp_wq;static const struct blk_mq_ops nvme_tcp_mq_ops;static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -3234,6 +3235,30 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,return ERR_PTR (ret); } +static int nvme_tcp_netdev_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +struct net_device *ndev =netdev_notifier_info_to_dev (ptr); +struct nvme_tcp_ctrl *ctrl; + +switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq); +/* +* The associated controllers teardown has completed, +* ddp contexts were also torn down so we should be +* safe to continue... +*/ + } +return NOTIFY_DONE ; +} +static struct nvmf_transport_ops nvme_tcp_transport = { .name ="tcp" , .module =THIS_MODULE ,
@@ -3248,6 +3273,8 @@ static struct nvmf_transport_ops nvme_tcp_transport = {static int __init nvme_tcp_init_module (void ) { +int ret; +BUILD_BUG_ON (sizeof (struct nvme_tcp_hdr ) !=8 );BUILD_BUG_ON (sizeof (struct nvme_tcp_cmd_pdu ) !=72 );BUILD_BUG_ON (sizeof (struct nvme_tcp_data_pdu ) !=24 );
@@ -3262,8 +3289,19 @@ static int __init nvme_tcp_init_module(void)if (!nvme_tcp_wq)return -ENOMEM ; + nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event; + ret =register_netdevice_notifier (&nvme_tcp_netdevice_nb); +if (ret) { +pr_err ("failed to register netdev notifier\n" ); + goto out_free_workqueue; + } +nvmf_register_transport (&nvme_tcp_transport);return 0 ; + +out_free_workqueue: +destroy_workqueue (nvme_tcp_wq); +return ret; } static void __exit nvme_tcp_cleanup_module(void)
@@ -3271,6 +3309,7 @@ static void __exit nvme_tcp_cleanup_module(void)struct nvme_tcp_ctrl *ctrl;nvmf_unregister_transport (&nvme_tcp_transport); +unregister_netdevice_notifier (&nvme_tcp_netdevice_nb);mutex_lock (&nvme_tcp_ctrl_mutex);list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list)
Signature · 1 line
2.34.1
Re: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 52 lines · Aurelien Aptel
From: Or Gerlitz <[email protected]>
For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ.
These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroys them.
Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo.
We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set.
This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister.
Signed-off-by Or Gerlitz <[email protected]>Signed-off-by Boris Pismenny <[email protected]>Signed-off-by Ben Ben-Ishay <[email protected]>Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>--- drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)drivers/nvme/host/tcp.c +6
-0@@ -221,6 +221,7 @@ struct nvme_tcp_ctrl {static LIST_HEAD (nvme_tcp_ctrl_list);static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); +static struct notifier_block nvme_tcp_netdevice_nb;static struct workqueue_struct *nvme_tcp_wq;static const struct blk_mq_ops nvme_tcp_mq_ops;static const struct blk_mq_ops nvme_tcp_admin_mq_ops;@@ -3234,6 +3235,30 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,return ERR_PTR (ret); } +static int nvme_tcp_netdev_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +struct net_device *ndev =netdev_notifier_info_to_dev (ptr); +struct nvme_tcp_ctrl *ctrl;
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 3252
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ +switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);
In what context is this called? because every time we flush a workqueue, lockdep finds another reason to complain about something...
Otherwise looks good,
Re: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 10 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 3252
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);
Quoted 2 lines
In what context is this called? because every time we flush a workqueue, lockdep finds another reason to complain about something...
Thanks for highlighting this, we re-checked it and we found that we are covered by nvme_tcp_error_recovery(), we can remove the flush_workqueue() call above.
Quoted 2 lines
Otherwise looks good,
Reviewed-by Sagi Grimberg <[email protected]>
Thanks
Sagi Grimberg
maintainerRe: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-0+switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);
Quoted 6 lines
Quoted 2 lines
In what context is this called? because every time we flush a workqueue, lockdep finds another reason to complain about something...
Thanks for highlighting this, we re-checked it and we found that we are covered by
nvme_tcp_error_recovery(), we can remove theflush_workqueue()call above.
Don't you need to flush at least err_work? How do you know that it completed and put all the references?
Re: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 13 lines · Sagi Grimberg
Quoted 13 lines
Quoted 12 lines
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 3252
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);In what context is this called? because every time we flush a workqueue, lockdep finds another reason to complain about something...
Quoted 6 lines
Quoted 3 lines
Thanks for highlighting this, we re-checked it and we found that we are covered by
nvme_tcp_error_recovery(), we can remove theflush_workqueue()call above.Don't you need to flush at least
err_work? How do you know that it completed and put all the references?
Our bad, we do need to wait for the netdev reference to be put, and we must keep the flush_workqueue().
We did test with lockdep but did not notice any warnings.
As for the context of the event handler when you set the link down is the process issuing the netlink syscall.
So if you run "ip link set X down" it would be (simplified):
"ip" -> syscall -> netlink api -> ... -> do_setlink -> call_netdevice_notifiers_info.
Sagi Grimberg
maintainerRe: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
drivers/nvme/host/tcp.c static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, · line 3252
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+switch (event) { +case NETDEV_GOING_DOWN : +mutex_lock (&nvme_tcp_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_tcp_ctrl_list, list) { +if (ndev == ctrl->offloading_netdev) +nvme_tcp_error_recovery (&ctrl->ctrl); + } +mutex_unlock (&nvme_tcp_ctrl_mutex); +flush_workqueue (nvme_reset_wq);Quoted 2 lines
In what context is this called? because every time we flush a workqueue, lockdep finds another reason to complain about something...
Thanks for highlighting this, we re-checked it and we found that we are covered by
nvme_tcp_error_recovery(), we can remove theflush_workqueue()call above.
Quoted 7 lines
Quoted 2 lines
Don't you need to flush at least
err_work? How do you know that it completed and put all the references?Our bad, we do need to wait for the netdev reference to be put, and we must keep the
flush_workqueue().We did test with lockdep but did not notice any warnings.
I'm assuming you are running with lockdep and friends?
Quoted 7 lines
As for the context of the event handler when you set the link down is the process issuing the netlink syscall.
So if you run "ip link set X down" it would be (simplified):
"ip" -> syscall -> netlink api -> ... ->
do_setlink->call_netdevice_notifiers_info.
ok, that should be fine I think.
Re: [PATCH v12 10/26] nvme-tcp: Deal with netdevice DOWN events
Quoted 1 line · Sagi Grimberg
I'm assuming you are running with lockdep and friends?
We tested it under the following configuration:
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RAW_LOCK_NESTING=y
CONFIG_LOCKDEP=y
CONFIG_LOCKDEP_BITS=15
CONFIG_LOCKDEP_CHAINS_BITS=16
CONFIG_LOCKDEP_STACK_TRACE_BITS=19
CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
CONFIG_DEBUG_LOCKDEP=y
CONFIG_PROVE_RCU=y
No warnings were printed in dmesg.
Anything else we should verify?
PATCH v12 11/26 nvme-tcp: Add modparam to control the ULP offload enablement
In reply to Aurelien AptelAdd ulp_offload module parameter to the nvme-tcp module to control ULP offload at the NVMe-TCP layer.
Turn ULP offload off be default, regardless of the NIC driver support.
Overall, in order to enable ULP offload:
- nvme-tcp ulp_offload modparam must be set to 1
- netdev->ulp_ddp_caps.active must have ULP_DDP_C_NVME_TCP and/or ULP_DDP_C_NVME_TCP_DDGST_RX capabilities flag set.
--- drivers/nvme/host/tcp.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +11 -1
@@ -49,6 +49,16 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)");#endif +#ifdef CONFIG_ULP_DDP +/* NVMeTCP direct data placement and data digest offload will not +* happen if this parameter false (default), regardless of what the +* underlying netdev capabilities are. +*/ +static bool ulp_offload; +module_param (ulp_offload, bool,0644 ); +MODULE_PARM_DESC (ulp_offload,"Enable or disable NVMeTCP ULP support" ); +#endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC /* lockdep can detect a circular dependency of the form * sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
@@ -350,7 +360,7 @@ static bool nvme_tcp_ddp_query_limits(struct net_device *netdev,static inline bool is_netdev_ulp_offload_active (struct net_device *netdev,struct nvme_tcp_queue *queue) { -if (!netdev || !queue) +if (!ulp_offload || !netdev || !queue)return false;/* If we cannot query the netdev limitations, do not offload */
Signature · 1 line
2.34.1
Sagi Grimberg
maintainerRe: [PATCH v12 11/26] nvme-tcp: Add modparam to control the ULP offload enablement
Quoted 23 lines · Aurelien Aptel
Add
ulp_offloadmodule parameter to the nvme-tcp module to control ULP offload at the NVMe-TCP layer.Turn ULP offload off be default, regardless of the NIC driver support.
Overall, in order to enable ULP offload:
- nvme-tcpulp_offloadmodparam must be set to 1
-netdev->ulp_ddp_caps.activemust haveULP_DDP_C_NVME_TCPand/orULP_DDP_C_NVME_TCP_DDGST_RXcapabilities flag set.Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>--- drivers/nvme/host/tcp.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -49,6 +49,16 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)");
drivers/nvme/host/tcp.c MODULE_PARM_DESC(tls_handshake_timeout, · line 59
Quoted 10 lines
drivers/nvme/host/tcp.c +8
-0#endif +#ifdef CONFIG_ULP_DDP +/* NVMeTCP direct data placement and data digest offload will not +* happen if this parameter false (default), regardless of what the +* underlying netdev capabilities are. +*/ +static bool ulp_offload; +module_param (ulp_offload, bool,0644 ); +MODULE_PARM_DESC (ulp_offload,"Enable or disable NVMeTCP ULP support" );
the name is strange.
maybe call it ddp_offload?
and in the description spell it as "direct data placement"
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-0+#endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC /* lockdep can detect a circular dependency of the form * sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
drivers/nvme/host/tcp.c static bool nvme_tcp_ddp_query_limits(struct net_device *netdev, · line 366
Quoted 8 lines
drivers/nvme/host/tcp.c +1
-1@@ -350,7 +360,7 @@ static bool nvme_tcp_ddp_query_limits(struct net_device *netdev,static inline bool is_netdev_ulp_offload_active (struct net_device *netdev,struct nvme_tcp_queue *queue) { -if (!netdev || !queue) +if (!ulp_offload || !netdev || !queue)return false;/* If we cannot query the netdev limitations, do not offload */
This patch should be folded to the control path. No reason for it to stand on its own I think.
Re: [PATCH v12 11/26] nvme-tcp: Add modparam to control the ULP offload enablement
Quoted 7 lines · Sagi Grimberg
drivers/nvme/host/tcp.c MODULE_PARM_DESC(tls_handshake_timeout, · line 59
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+static bool ulp_offload; +module_param (ulp_offload, bool,0644 ); +MODULE_PARM_DESC (ulp_offload,"Enable or disable NVMeTCP ULP support" );the name is strange.
maybe call itddp_offload?
and in the description spell it as "direct data placement"
Sure.
Quoted 2 lines
This patch should be folded to the control path. No reason for it to stand on its own I think.
Sure, we will fold it.
Thanks
Re: [PATCH v12 11/26] nvme-tcp: Add modparam to control the ULP offload enablement
In reply to Sagi GrimbergQuoted 30 lines · Sagi Grimberg
Quoted 27 lines · Aurelien Aptel
Add
ulp_offloadmodule parameter to the nvme-tcp module to control ULP offload at the NVMe-TCP layer.Turn ULP offload off be default, regardless of the NIC driver support.
Overall, in order to enable ULP offload:
- nvme-tcpulp_offloadmodparam must be set to 1
-netdev->ulp_ddp_caps.activemust haveULP_DDP_C_NVME_TCPand/orULP_DDP_C_NVME_TCP_DDGST_RXcapabilities flag set.Signed-off-by Yoray Zack <[email protected]>Signed-off-by Shai Malin <[email protected]>Signed-off-by Aurelien Aptel <[email protected]>---
drivers/nvme/host/tcp.c| 12 +++++++++++-1 file changed, 11 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +3
-0@@ -49,6 +49,16 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)"); #endif +#ifdef CONFIG_ULP_DDP +/* NVMeTCP direct data placement and data digest offload will not +* happen if this parameter false (default), regardless of what the
Quoted 8 lines
drivers/nvme/host/tcp.c MODULE_PARM_DESC(tls_handshake_timeout, · line 59
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+* underlying netdev capabilities are. +*/ +static bool ulp_offload; +module_param (ulp_offload, bool,0644 ); +MODULE_PARM_DESC (ulp_offload,"Enable or disable NVMeTCP ULP support" );the name is strange.
maybe call itddp_offload?
Agree. ddp_offload is a better fit.
Quoted 11 lines
and in the description spell it as "direct data placement"
Quoted 9 lines
drivers/nvme/host/tcp.c +2
-0+#endif + #ifdef CONFIG_DEBUG_LOCK_ALLOC /* lockdep can detect a circular dependency of the form* sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock @@ -350,7 +360,7 @@ static bool nvme_tcp_ddp_query_limits(structnet_device *netdev, static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, struct nvme_tcp_queue *queue)
Quoted 8 lines
drivers/nvme/host/tcp.c static bool nvme_tcp_ddp_query_limits(struct net_device *netdev, · line 366
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-1{ - if (!netdev || !queue) + if (!ulp_offload || !netdev || !queue) return false; /* If we cannot query the netdev limitations, do not offload */This patch should be folded to the control path. No reason for it to stand on its own I think.
Agree.
PATCH v12 12/26 nvme-tcp: Only enable offload with TLS if the driver supports it
In reply to Aurelien AptelCheck if ULP offload driver supports ULP-over-TLS before enabling the offload with tls.
--- drivers/nvme/host/tcp.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/nvme/host/tcp.c +4 -0
@@ -367,6 +367,10 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,if (!nvme_tcp_ddp_query_limits (netdev, queue))return false; +/* If we are using TLS and netdev doesn't support it, do not offload */ +if (queue->ctrl->ctrl.opts->tls && !queue->ddp_limits.tls) +return false; +/* If netdev supports nvme-tcp ddp offload, we can offload */ if (test_bit (ULP_DDP_C_NVME_TCP_BIT , netdev->ulp_ddp_caps.active))return true;
Signature · 1 line
2.34.1
Sagi Grimberg
maintainerRe: [PATCH v12 12/26] nvme-tcp: Only enable offload with TLS if the driver supports it
Quoted 15 lines · Aurelien Aptel
Check if ULP offload driver supports ULP-over-TLS before enabling the offload with tls.
Signed-off-by Aurelien Aptel <[email protected]>Signed-off-by Shai Malin <[email protected]>Reviewed-by Max Gurtovoy <[email protected]>--- drivers/nvme/host/tcp.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 372
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0@@ -367,6 +367,10 @@ static inline bool is_netdev_ulp_offload_active(struct net_device *netdev,if (!nvme_tcp_ddp_query_limits (netdev, queue))return false; +/* If we are using TLS and netdev doesn't support it, do not offload */ +if (queue->ctrl->ctrl.opts->tls && !queue->ddp_limits.tls) +return false;
Same for this, fold to the first patch.
Other than that I had a question on one of my other responses. I don't think that tls_device supports 1.3, so what does tls here mean? That any device that enables this supports _all_ tls versions?
Re: [PATCH v12 12/26] nvme-tcp: Only enable offload with TLS if the driver supports it
Quoted 5 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static inline bool is_netdev_ulp_offload_active(struct net_device *netdev, · line 372
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+/* If we are using TLS and netdev doesn't support it, do not offload */ +if (queue->ctrl->ctrl.opts->tls && !queue->ddp_limits.tls) +return false;Same for this, fold to the first patch.
Ok
Quoted 4 lines
Other than that I had a question on one of my other responses. I don't think that
tls_devicesupports 1.3, so what does tls here mean? That any device that enables this supports _all_ tls versions?
This is a catch-all for all TLS versions.
We can split the logic between TLS version once a HW device will allow it.
Thanks
PATCH v12 13/26 Documentation: add ULP DDP offload documentation
In reply to Aurelien AptelFrom: Yoray Zack <[email protected]>
Document the new ULP DDP API and add it under "networking". Use NVMe-TCP implementation as an example.
--- Documentation/networking/index.rst | 1 + Documentation/networking/ulp-ddp-offload.rst | 378 +++++++++++++++++++ 2 files changed, 379 insertions(+) create mode 100644 Documentation/networking/ulp-ddp-offload.rst
Documentation/networking/index.rst +1 -0
@@ -108,6 +108,7 @@ Contents: tc-actions-env-rules tc-queue-filters tcp-thin + ulp-ddp-offload team timestamping tipc
Documentation/networking/ulp-ddp-offload.rst new file +378 -0
378 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v12 13/26] Documentation: add ULP DDP offload documentation
Quoted 4 lines · Aurelien Aptel
From: Yoray Zack <[email protected]>
Document the new ULP DDP API and add it under "networking". Use NVMe-TCP implementation as an example.
...
Quoted 5 lines
Documentation/networking/ulp-ddp-offload.rst new file +0
-0
...
Documentation/networking/ulp-ddp-offload.rst line 74
Quoted 9 lines
Documentation/networking/ulp-ddp-offload.rst new file +9
-0+Device configuration +==================== + +During driver initialization the driver sets the following +:c:type:`struct net_device <net_device>` properties: + +* The ULP DDP capabilities it supports + in :c:type:`struct ulp_ddp_netdev_caps <ulp_ddp_caps>` +* The ULP DDP operations pointer in :c:type:`struct ulp_ddp_dev_ops`.
'make htmldocs' seems a little unhappy about this for some reason:
.../ulp-ddp-offload.rst:74: WARNING: Unparseable C cross-reference: 'struct ulp_ddp_dev_ops' Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
struct ulp_ddp_dev_ops
------^Quoted 16 lines
Documentation/networking/ulp-ddp-offload.rst new file +16
-0+ +The current list of capabilities is represented as a bitset: + +.. code-block:: c + + enum { + ULP_DDP_C_NVME_TCP_BIT, + ULP_DDP_C_NVME_TCP_DDGST_RX_BIT, + /* add capabilities above */ + ULP_DDP_C_COUNT, + }; + +The enablement of capabilities can be controlled from userspace via +netlink. See Documentation/networking/ethtool-netlink.rst for more +details. +
Documentation/networking/ulp-ddp-offload.rst line 100
Quoted 10 lines
Documentation/networking/ulp-ddp-offload.rst new file +10
-0+Later, after the L5P completes its handshake, the L5P queries the +driver for its runtime limitations via the :c:member:`limits` operation: + +.. code-block:: c + + int (*limits)(struct net_device *netdev, + struct ulp_ddp_limits *lim); + + +All L5P share a common set of limits and parameters (:c:type:`struct ulp_ddp_limits`):
Likewise, here:
.../ulp-ddp-offload.rst:100: WARNING: Unparseable C cross-reference: 'struct ulp_ddp_limits' Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
struct ulp_ddp_limits
------^Quoted 24 lines
Documentation/networking/ulp-ddp-offload.rst new file +24
-0+ +.. code-block:: c + + /** + * struct ulp_ddp_limits - Generic ulp ddp limits: tcp ddp + * protocol limits. + * Add new instances of ulp_ddp_limits in the union below (nvme-tcp, etc.). + * + * @type: type of this limits struct + * @max_ddp_sgl_len: maximum sgl size supported (zero means no limit) + * @io_threshold: minimum payload size required to offload + * @tls: support for ULP over TLS + * @nvmeotcp: NVMe-TCP specific limits + */ + struct ulp_ddp_limits { + enum ulp_ddp_type type; + int max_ddp_sgl_len; + int io_threshold; + bool tls:1; + union { + /* ... protocol-specific limits ... */ + struct nvme_tcp_ddp_limits nvmeotcp; + }; + };
...
Documentation/networking/ulp-ddp-offload.rst line 288
Quoted 9 lines
Documentation/networking/ulp-ddp-offload.rst new file +9
-0+Asynchronous teardown +--------------------- + +To teardown the association between tags and buffers and allow tag reuse NIC HW +is called by the NIC driver during `teardown`. This operation may be +performed either synchronously or asynchronously. In asynchronous teardown, +`teardown` returns immediately without unmapping NIC HW buffers. Later, +when the unmapping completes by NIC HW, the NIC driver will call up to L5P +using :c:member:`ddp_teardown_done` of :c:type:`struct ulp_ddp_ulp_ops`:
And here:
.../ulp-ddp-offload.rst:283: WARNING: Unparseable C cross-reference: 'struct ulp_ddp_ulp_ops' Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
struct ulp_ddp_ulp_ops
------^Quoted 26 lines
Documentation/networking/ulp-ddp-offload.rst new file +26
-0+ +.. code-block:: c + + void (*ddp_teardown_done)(void *ddp_ctx); + +The `ddp_ctx` parameter passed in `ddp_teardown_done` is the same on provided +in `teardown` and it is used to carry some context about the buffers +and tags that are released. + +Resync handling +=============== + +RX +-- +In presence of packet drops or network packet reordering, the device may lose +synchronization between the TCP stream and the L5P framing, and require a +resync with the kernel's TCP stack. When the device is out of sync, no offload +takes place, and packets are passed as-is to software. Resync is very similar +to TLS offload (see documentation at Documentation/networking/tls-offload.rst) + +If only packets with L5P data are lost or reordered, then resynchronization may +be avoided by NIC HW that keeps tracking PDU headers. If, however, PDU headers +are reordered, then resynchronization is necessary. + +To resynchronize hardware during traffic, we use a handshake between hardware +and software. The NIC HW searches for a sequence of bytes that identifies L5P
Documentation/networking/ulp-ddp-offload.rst line 324
Quoted 10 lines
Documentation/networking/ulp-ddp-offload.rst new file +10
-0+headers (i.e., magic pattern). For example, in NVMe-TCP, the PDU operation +type can be used for this purpose. Using the PDU header length field, the NIC +HW will continue to find and match magic patterns in subsequent PDU headers. If +the pattern is missing in an expected position, then searching for the pattern +starts anew. + +The NIC will not resume offload when the magic pattern is first identified. +Instead, it will request L5P software to confirm that indeed this is a PDU +header. To request confirmation the NIC driver calls up to L5P using +:c:member:`*resync_request` of :c:type:`struct ulp_ddp_ulp_ops`:
And here:
.../ulp-ddp-offload.rst:321: WARNING: Unparseable C cross-reference: '*resync_request' Invalid C declaration: Expected identifier in nested name. [error at 0]
*resync_request
^
.../ulp-ddp-offload.rst:321: WARNING: Unparseable C cross-reference: 'struct ulp_ddp_ulp_ops'
Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
struct ulp_ddp_ulp_ops
------^Quoted 19 lines
Documentation/networking/ulp-ddp-offload.rst new file +19
-0+ +.. code-block:: c + + bool (*resync_request)(struct sock *sk, u32 seq, u32 flags); + +The `seq` parameter contains the TCP sequence of the last byte in the PDU header. +The `flags` parameter contains a flag (`ULP_DDP_RESYNC_PENDING`) indicating whether +a request is pending or not. +L5P software will respond to this request after observing the packet containing +TCP sequence `seq` in-order. If the PDU header is indeed there, then L5P +software calls the NIC driver using the :c:member:`resync` function of +the :c:type:`struct ulp_ddp_dev_ops <ulp_ddp_ops>` inside the :c:type:`struct +net_device <net_device>` while passing the same `seq` to confirm it is a PDU +header. + +.. code-block:: c + + void (*resync)(struct net_device *netdev, + struct sock *sk, u32 seq);
...
Re: [PATCH v12 13/26] Documentation: add ULP DDP offload documentation
Quoted 6 lines · Simon Horman
'make htmldocs' seems a little unhappy about this for some reason:
.../ulp-ddp-offload.rst:74: WARNING: Unparseable C cross-reference: 'structulp_ddp_dev_ops' Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]struct ulp_ddp_dev_ops ------^
I've fixed all the warnings by turning
:c:type:struct foo
into
:c:type:`struct foo <foo>`
Thanks
PATCH v12 14/26 net/mlx5e: Rename from tls to transport static params
In reply to Aurelien AptelFrom: Or Gerlitz <[email protected]>
The static params structure is used in TLS but also in other transports we're offloading like nvmeotcp:
- Rename the relevant structures/fields
- Create common file for appropriate transports
- Apply changes in the TLS code
No functional change here.
--- .../mlx5/core/en_accel/common_utils.h | 32 +++++++++++++++++ .../mellanox/mlx5/core/en_accel/ktls.c | 2 +- .../mellanox/mlx5/core/en_accel/ktls_rx.c | 6 ++-- .../mellanox/mlx5/core/en_accel/ktls_tx.c | 8 ++--- .../mellanox/mlx5/core/en_accel/ktls_txrx.c | 36 ++++++++----------- .../mellanox/mlx5/core/en_accel/ktls_utils.h | 17 ++------- include/linux/mlx5/device.h | 8 ++--- include/linux/mlx5/mlx5_ifc.h | 8 +++-- 8 files changed, 67 insertions(+), 50 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */ +#ifndef __MLX5E_COMMON_UTILS_H__ +#define __MLX5E_COMMON_UTILS_H__ + +#include "en.h" + +struct mlx5e_set_transport_static_params_wqe { +struct mlx5_wqe_ctrl_seg ctrl; +struct mlx5_wqe_umr_ctrl_seg uctrl; +struct mlx5_mkey_seg mkc; +struct mlx5_wqe_transport_static_params_seg params; +}; + +/* macros for transport_static_params handling */ +#define MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS \ + (DIV_ROUND_UP(sizeof(struct mlx5e_set_transport_static_params_wqe), MLX5_SEND_WQE_BB)) + +#define MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE (sq, pi) \ + ((struct mlx5e_set_transport_static_params_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_transport_static_params_wqe))) + +#define MLX5E_TRANSPORT_STATIC_PARAMS_WQE_SZ \ + (sizeof(struct mlx5e_set_transport_static_params_wqe)) + +#define MLX5E_TRANSPORT_STATIC_PARAMS_DS_CNT \ + (DIV_ROUND_UP(MLX5E_TRANSPORT_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS)) + +#define MLX5E_TRANSPORT_STATIC_PARAMS_OCTWORD_SIZE \ + (MLX5_ST_SZ_BYTES(transport_static_params) / MLX5_SEND_WQE_DS) + +#endif /* __MLX5E_COMMON_UTILS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c +1 -1
@@ -100,7 +100,7 @@ bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev)return false;/* Check the possibility to post the required ICOSQ WQEs. */ -if (WARN_ON_ONCE (max_sq_wqebbs <MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS )) +if (WARN_ON_ONCE (max_sq_wqebbs <MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS ))return false;if (WARN_ON_ONCE (max_sq_wqebbs <MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS ))return false;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +3 -3
@@ -136,16 +136,16 @@ static struct mlx5_wqe_ctrl_seg * post_static_params(struct mlx5e_icosq *sq, struct mlx5e_ktls_offload_context_rx *priv_rx) { - struct mlx5e_set_tls_static_params_wqe *wqe; + struct mlx5e_set_transport_static_params_wqe *wqe;struct mlx5e_icosq_wqe_info wi;u16 pi, num_wqebbs; - num_wqebbs =MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS ; + num_wqebbs =MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS ;if (unlikely (!mlx5e_icosq_can_post_wqe (sq, num_wqebbs)))return ERR_PTR (-ENOSPC ); pi =mlx5e_icosq_get_next_pi (sq, num_wqebbs); - wqe =MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE (sq, pi); + wqe =MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE (sq, pi); mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_rx->crypto_info, mlx5e_tir_get_tirn(&priv_rx->tir), mlx5_crypto_dek_get_id(priv_rx->dek),
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +4 -4
@@ -33,7 +33,7 @@ u16 mlx5e_ktls_get_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *pa num_dumps =mlx5e_ktls_dumps_num_wqes (params,MAX_SKB_FRAGS ,TLS_MAX_PAYLOAD_SIZE ); - stop_room +=mlx5e_stop_room_for_wqe (mdev,MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS ); + stop_room +=mlx5e_stop_room_for_wqe (mdev,MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS ); stop_room +=mlx5e_stop_room_for_wqe (mdev,MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS ); stop_room += num_dumps *mlx5e_stop_room_for_wqe (mdev,MLX5E_KTLS_DUMP_WQEBBS ); stop_room +=1 ;/* fence nop */
@@ -550,12 +550,12 @@ post_static_params(struct mlx5e_txqsq *sq, struct mlx5e_ktls_offload_context_tx *priv_tx, bool fence) { -struct mlx5e_set_tls_static_params_wqe *wqe; +struct mlx5e_set_transport_static_params_wqe *wqe;u16 pi, num_wqebbs; - num_wqebbs =MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS ; + num_wqebbs =MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS ; pi =mlx5e_txqsq_get_next_pi (sq, num_wqebbs); - wqe =MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE (sq, pi); + wqe =MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE (sq, pi); mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_tx->crypto_info, priv_tx->tisn, mlx5_crypto_dek_get_id(priv_tx->dek),
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c +15 -21
@@ -8,10 +8,6 @@ enum { MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2, }; -enum { -MLX5E_ENCRYPTION_STANDARD_TLS =0x1 , -}; -#define EXTRACT_INFO_FIELDS do { \ salt = info->salt; \ rec_seq = info->rec_seq; \
@@ -20,7 +16,7 @@ enum { }while (0 ) static void -fill_static_params (struct mlx5_wqe_tls_static_params_seg *params, +fill_static_params (struct mlx5_wqe_transport_static_params_seg *params,union mlx5e_crypto_info *crypto_info,u32 key_id,u32 resync_tcp_sn) {
@@ -53,25 +49,25 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,return ; } - gcm_iv =MLX5_ADDR_OF (tls_static_params, ctx, gcm_iv); - initial_rn =MLX5_ADDR_OF (tls_static_params, ctx, initial_record_number); + gcm_iv =MLX5_ADDR_OF (transport_static_params, ctx, gcm_iv); + initial_rn =MLX5_ADDR_OF (transport_static_params, ctx, initial_record_number);memcpy (gcm_iv, salt, salt_sz);memcpy (initial_rn, rec_seq, rec_seq_sz); tls_version =MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 ; -MLX5_SET (tls_static_params, ctx, tls_version, tls_version); -MLX5_SET (tls_static_params, ctx, const_1,1 ); -MLX5_SET (tls_static_params, ctx, const_2,2 ); -MLX5_SET (tls_static_params, ctx, encryption_standard, -MLX5E_ENCRYPTION_STANDARD_TLS ); -MLX5_SET (tls_static_params, ctx, resync_tcp_sn, resync_tcp_sn); -MLX5_SET (tls_static_params, ctx, dek_index, key_id); +MLX5_SET (transport_static_params, ctx, tls_version, tls_version); +MLX5_SET (transport_static_params, ctx, const_1,1 ); +MLX5_SET (transport_static_params, ctx, const_2,2 ); +MLX5_SET (transport_static_params, ctx, acc_type, +MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS ); +MLX5_SET (transport_static_params, ctx, resync_tcp_sn, resync_tcp_sn); +MLX5_SET (transport_static_params, ctx, dek_index, key_id); } void -mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe, +mlx5e_ktls_build_static_params(struct mlx5e_set_transport_static_params_wqe *wqe, u16 pc, u32 sqn, union mlx5e_crypto_info *crypto_info, u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
@@ -80,19 +76,17 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;u8 opmod = direction ==TLS_OFFLOAD_CTX_DIR_TX ? -MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS : -MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS ; - -#define STATIC_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS) +MLX5_OPC_MOD_TRANSPORT_TIS_STATIC_PARAMS : +MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS ; cseg->opmod_idx_opcode =cpu_to_be32 ((pc <<8 ) |MLX5_OPCODE_UMR | (opmod <<24 )); cseg->qpn_ds =cpu_to_be32 ((sqn <<MLX5_WQE_CTRL_QPN_SHIFT ) | -STATIC_PARAMS_DS_CNT ); +MLX5E_TRANSPORT_STATIC_PARAMS_DS_CNT ); cseg->fm_ce_se = fence ?MLX5_FENCE_MODE_INITIATOR_SMALL :0 ; cseg->tis_tir_num =cpu_to_be32 (tis_tir_num <<8 ); ucseg->flags =MLX5_UMR_INLINE ; - ucseg->bsf_octowords =cpu_to_be16 (MLX5_ST_SZ_BYTES (tls_static_params) /16 ); + ucseg->bsf_octowords =cpu_to_be16 (MLX5E_TRANSPORT_STATIC_PARAMS_OCTWORD_SIZE );fill_static_params (&wqe->params, crypto_info, key_id, resync_tcp_sn); }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h +2 -15
@@ -6,6 +6,7 @@#include <net/tls.h> #include "en.h" +#include "en_accel/common_utils.h" enum {MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD =0 ,
@@ -33,13 +34,6 @@ union mlx5e_crypto_info {struct tls12_crypto_info_aes_gcm_256 crypto_info_256; }; -struct mlx5e_set_tls_static_params_wqe { -struct mlx5_wqe_ctrl_seg ctrl; -struct mlx5_wqe_umr_ctrl_seg uctrl; -struct mlx5_mkey_seg mkc; -struct mlx5_wqe_tls_static_params_seg params; -}; -struct mlx5e_set_tls_progress_params_wqe {struct mlx5_wqe_ctrl_seg ctrl;struct mlx5_wqe_tls_progress_params_seg params;
@@ -50,19 +44,12 @@ struct mlx5e_get_tls_progress_params_wqe {struct mlx5_seg_get_psv psv; }; -#define MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS \ - (DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_static_params_wqe), MLX5_SEND_WQE_BB)) -#define MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS \ (DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_progress_params_wqe), MLX5_SEND_WQE_BB))#define MLX5E_KTLS_GET_PROGRESS_WQEBBS \ (DIV_ROUND_UP(sizeof(struct mlx5e_get_tls_progress_params_wqe), MLX5_SEND_WQE_BB)) -#define MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE (sq, pi) \ - ((struct mlx5e_set_tls_static_params_wqe *)\ - mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_static_params_wqe))) -#define MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE (sq, pi) \ ((struct mlx5e_set_tls_progress_params_wqe *)\ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_progress_params_wqe)))
@@ -76,7 +63,7 @@ struct mlx5e_get_tls_progress_params_wqe { mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe))) void -mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe, +mlx5e_ktls_build_static_params(struct mlx5e_set_transport_static_params_wqe *wqe, u16 pc, u32 sqn, union mlx5e_crypto_info *crypto_info, u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
include/linux/mlx5/device.h +4 -4
@@ -449,8 +449,8 @@ enum { };enum { -MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS =0x1 , -MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS =0x2 , +MLX5_OPC_MOD_TRANSPORT_TIS_STATIC_PARAMS =0x1 , +MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS =0x2 , };enum {
@@ -458,8 +458,8 @@ enum { MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS = 0x2, }; -struct mlx5_wqe_tls_static_params_seg { -u8 ctx[MLX5_ST_SZ_BYTES (tls_static_params)]; +struct mlx5_wqe_transport_static_params_seg { +u8 ctx[MLX5_ST_SZ_BYTES (transport_static_params)]; };struct mlx5_wqe_tls_progress_params_seg {
include/linux/mlx5/mlx5_ifc.h +6 -2
@@ -12314,12 +12314,16 @@ enum { MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_PURPOSE_MACSEC = 0x4, }; -struct mlx5_ifc_tls_static_params_bits { +enum { +MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS =0x1 , +}; + +struct mlx5_ifc_transport_static_params_bits {u8 const_2[0x2 ];u8 tls_version[0x4 ];u8 const_1[0x2 ];u8 reserved_at_8[0x14 ]; -u8 encryption_standard[0x4 ]; +u8 acc_type[0x4 ];u8 reserved_at_20[0x20 ];
Signature · 1 line
2.34.1
PATCH v12 15/26 net/mlx5e: Refactor ico sq polling to get budget
In reply to Aurelien AptelFrom: Or Gerlitz <[email protected]>
The mlx5e driver uses ICO SQs for internal control operations which are not visible to the network stack, such as UMR mapping for striding RQ (MPWQ) and etc more cases.
The upcoming nvmeotcp offload uses ico sq for umr mapping as part of the offload. As a pre-step for nvmeotcp ico sqs which have their own napi and need to comply with budget, add the budget as parameter to the polling of cqs related to ico sqs.
The polling already stops after a limit is reached, so just have the caller to provide this limit as the budget.
No functional change here.
--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 4 ++-- drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +1 -1
@@ -62,7 +62,7 @@ void mlx5e_trigger_irq(struct mlx5e_icosq *sq);void mlx5e_completion_event (struct mlx5_core_cq *mcq,struct mlx5_eqe *eqe);void mlx5e_cq_error_event (struct mlx5_core_cq *mcq,enum mlx5_event event);int mlx5e_napi_poll (struct napi_struct *napi,int budget); -int mlx5e_poll_ico_cq (struct mlx5e_cq *cq); +int mlx5e_poll_ico_cq (struct mlx5e_cq *cq,int budget);/* RX */ INDIRECT_CALLABLE_DECLARE(bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq));
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +2 -2
@@ -968,7 +968,7 @@ static void mlx5e_handle_shampo_hd_umr(struct mlx5e_shampo_umr umr, shampo->ci = (shampo->ci + umr.len) & (shampo->hd_per_wq -1 ); } -int mlx5e_poll_ico_cq (struct mlx5e_cq *cq) +int mlx5e_poll_ico_cq (struct mlx5e_cq *cq,int budget) { struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);struct mlx5_cqe64 *cqe;
@@ -1043,7 +1043,7 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq) wi->wqe_type); } }while (!last_wqe); - }while ((++i <MLX5E_TX_CQ_POLL_BUDGET ) && (cqe =mlx5_cqwq_get_cqe (&cq->wq))); + }while ((++i < budget) && (cqe =mlx5_cqwq_get_cqe (&cq->wq))); sq->cc = sqcc;
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +2 -2
@@ -178,8 +178,8 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) busy |= work_done == budget; -mlx5e_poll_ico_cq (&c->icosq.cq); -if (mlx5e_poll_ico_cq (&c->async_icosq.cq)) +mlx5e_poll_ico_cq (&c->icosq.cq,MLX5E_TX_CQ_POLL_BUDGET ); +if (mlx5e_poll_ico_cq (&c->async_icosq.cq,MLX5E_TX_CQ_POLL_BUDGET ))/* Don't clear the flag if nothing was polled to prevent * queueing more WQEs and overflowing the async ICOSQ. */
Signature · 1 line
2.34.1
PATCH v12 16/26 net/mlx5e: Have mdev pointer directly on the icosq structure
In reply to Aurelien AptelFrom: Or Gerlitz <[email protected]>
This provides better separation between channels to ICO SQs for use-cases where they are not tightly coupled (such as the upcoming nvmeotcp code).
No functional change here.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c | 4 ++-- drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 5 ++--- 4 files changed, 6 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -547,6 +547,7 @@ struct mlx5e_icosq {/* control path */ struct mlx5_wq_ctrl wq_ctrl;struct mlx5e_channel *channel; +struct mlx5_core_dev *mdev;struct work_struct recover_work; }____cacheline_aligned_in_smp ;
drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c +2 -2
@@ -46,7 +46,7 @@ static int mlx5e_query_rq_state(struct mlx5_core_dev *dev, u32 rqn, u8 *state)static int mlx5e_wait_for_icosq_flush (struct mlx5e_icosq *icosq) { -struct mlx5_core_dev *dev = icosq->channel->mdev; +struct mlx5_core_dev *dev = icosq->mdev;unsigned long exp_time; exp_time = jiffies +msecs_to_jiffies (mlx5_tout_ms (dev,FLUSH_ON_ERROR ));
@@ -91,7 +91,7 @@ static int mlx5e_rx_reporter_err_icosq_cqe_recover(void *ctx) rq = &icosq->channel->rq;if (test_bit (MLX5E_RQ_STATE_ENABLED , &icosq->channel->xskrq.state)) xskrq = &icosq->channel->xskrq; - mdev = icosq->channel->mdev; + mdev = icosq->mdev; dev = icosq->channel->netdev; err =mlx5_core_query_sq_state (mdev, icosq->sqn, &state);if (err) {
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +1 -1
@@ -267,7 +267,7 @@ resync_post_get_progress_params(struct mlx5e_icosq *sq, goto err_out; } - pdev =mlx5_core_dma_dev (sq->channel->priv->mdev); + pdev =mlx5_core_dma_dev (sq->mdev); buf->dma_addr =dma_map_single (pdev, &buf->progress,PROGRESS_PARAMS_PADDED_SIZE ,DMA_FROM_DEVICE );if (unlikely (dma_mapping_error (pdev, buf->dma_addr))) {
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -3
@@ -1422,6 +1422,7 @@ static int mlx5e_alloc_icosq(struct mlx5e_channel *c,int err; sq->channel = c; + sq->mdev = mdev; sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map; sq->reserved_room = param->stop_room;
@@ -1820,11 +1821,9 @@ void mlx5e_deactivate_icosq(struct mlx5e_icosq *icosq)static void mlx5e_close_icosq (struct mlx5e_icosq *sq) { -struct mlx5e_channel *c = sq->channel; -if (sq->ktls_resync)mlx5e_ktls_rx_resync_destroy_resp_list (sq->ktls_resync); -mlx5e_destroy_sq (c->mdev, sq->sqn); +mlx5e_destroy_sq (sq->mdev, sq->sqn);mlx5e_free_icosq_descs (sq);mlx5e_free_icosq (sq); }
Signature · 1 line
2.34.1
PATCH v12 17/26 net/mlx5e: Refactor doorbell function to allow avoiding a completion
In reply to Aurelien AptelFrom: Or Gerlitz <[email protected]>
Currently the doorbell function always asks for completion to be generated.
Refactor things such that all existing call sites are untouched and no branching is added. This is done using inner function which can be invoked directly in cases completion is not desired (as done in downstream patch).
No functional change here.
--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +10 -3
@@ -256,10 +256,10 @@ static inline u16 mlx5e_icosq_get_next_pi(struct mlx5e_icosq *sq, u16 size) }static inline void -mlx5e_notify_hw (struct mlx5_wq_cyc *wq,u16 pc,void __iomem *uar_map, -struct mlx5_wqe_ctrl_seg *ctrl) +__mlx5e_notify_hw (struct mlx5_wq_cyc *wq,u16 pc,void __iomem *uar_map, +struct mlx5_wqe_ctrl_seg *ctrl,u8 cq_update) { - ctrl->fm_ce_se |=MLX5_WQE_CTRL_CQ_UPDATE ; + ctrl->fm_ce_se |= cq_update;/* ensure wqe is visible to device before updating doorbell record */ dma_wmb ();
@@ -273,6 +273,13 @@ mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,mlx5_write64 ((__be32 *)ctrl, uar_map); } +static inline void +mlx5e_notify_hw (struct mlx5_wq_cyc *wq,u16 pc,void __iomem *uar_map, +struct mlx5_wqe_ctrl_seg *ctrl) +{ +__mlx5e_notify_hw (wq, pc, uar_map, ctrl,MLX5_WQE_CTRL_CQ_UPDATE ); +} +static inline void mlx5e_cq_arm (struct mlx5e_cq *cq) {struct mlx5_core_cq *mcq;
Signature · 1 line
2.34.1
PATCH v12 18/26 net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
Add the necessary infrastructure for NVMEoTCP offload:
- Create mlx5_cqe128 structure for NVMEoTCP offload. The new structure consist from the regular mlx5_cqe64 + NVMEoTCP data information for offloaded packets.
- Add nvmetcp field to mlx5_cqe64, this field define the type of the data that the additional NVMEoTCP part represents.
- Add nvmeotcp_zero_copy_en + nvmeotcp_crc_en bit to the TIR, for identify NVMEoTCP offload flow and tag_buffer_id that will be used by the connected nvmeotcp_queues.
- Add new capability to HCA_CAP that represents the NVMEoTCP offload ability.
--- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++ include/linux/mlx5/device.h | 51 +++++++++++++- include/linux/mlx5/mlx5_ifc.h | 74 ++++++++++++++++++-- include/linux/mlx5/qp.h | 1 + 4 files changed, 127 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
@@ -289,6 +289,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)return err; } +if (MLX5_CAP_GEN (dev, nvmeotcp)) { + err =mlx5_core_get_caps (dev,MLX5_CAP_DEV_NVMEOTCP ); +if (err) +return err; + } +return 0 ; }
include/linux/mlx5/device.h +50 -1
87 lines not on this page · raw message
include/linux/mlx5/mlx5_ifc.h +70 -4
139 lines not on this page · raw message
include/linux/mlx5/qp.h +1 -0
@@ -227,6 +227,7 @@ struct mlx5_wqe_ctrl_seg {#define MLX5_WQE_CTRL_OPCODE_MASK 0xff#define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00#define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8 +#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8enum {MLX5_ETH_WQE_L3_INNER_CSUM =1 <<4 ,
Signature · 1 line
2.34.1
PATCH v12 19/26 net/mlx5e: NVMEoTCP, offload initialization
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
This commit introduces the driver structures and initialization blocks for NVMEoTCP offload. The mlx5 nvmeotcp structures are:
- queue (mlx5e_nvmeotcp_queue) - pairs 1:1 with nvmeotcp driver queues and deals with the offloading parts. The mlx5e queue is accessed in the ddp
ops: initialized on sk_add, used in ddp setup,teardown,resync and in the fast path when dealing with packets, destroyed in the sk_del op.
- queue entry (nvmeotcp_queue_entry) - pairs 1:1 with offloaded IO from that queue. Keeps pointers to the SG elements describing the buffers used for the IO and the ddp context of it.
- queue handler (mlx5e_nvmeotcp_queue_handler) - we use icosq per NVME-TCP queue for UMR mapping as part of the ddp offload. Those dedicated SQs are unique in the sense that they are driven directly by the NVME-TCP layer to submit and invalidate ddp requests.
Since the life-cycle of these icosqs is not tied to the channels, we create dedicated napi contexts for polling them such that channels can be re-created during offloading. The queue handler has pointer to the cq associated with the queue's sq and napi context.
- main offload context (mlx5e_nvmeotcp) - has ida and hash table instances. Each offloaded queue gets an ID from the ida instance and the <id, queue> pairs are kept in the hash table. The id is programmed as flow tag to be set by HW on the completion (cqe) of all packets related to this queue (by 5-tuple steering). The fast path which deals with packets uses the flow tag to access the hash table and retrieve the queue for the processing.
We query nvmeotcp capabilities to see if the offload can be supported and use 128B CQEs when this happens. By default, the offload is off but can be enabled with ethtool --ulp-ddp <device> nvme-tcp-ddp on.
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 11 + .../net/ethernet/mellanox/mlx5/core/Makefile | 2 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 + .../net/ethernet/mellanox/mlx5/core/en/fs.h | 4 +- .../ethernet/mellanox/mlx5/core/en/params.c | 12 +- .../ethernet/mellanox/mlx5/core/en/params.h | 3 + .../mellanox/mlx5/core/en_accel/en_accel.h | 3 + .../mellanox/mlx5/core/en_accel/fs_tcp.h | 2 +- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 216 ++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 121 ++++++++++ .../ethernet/mellanox/mlx5/core/en_ethtool.c | 6 + .../net/ethernet/mellanox/mlx5/core/en_fs.c | 4 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 17 ++ .../net/ethernet/mellanox/mlx5/core/main.c | 1 + 14 files changed, 397 insertions(+), 9 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +11 -0
@@ -163,6 +163,17 @@ config MLX5_EN_TLS help Build support for TLS cryptography-offload acceleration in the NIC. +config MLX5_EN_NVMEOTCP + bool "NVMEoTCP acceleration" + depends on ULP_DDP + depends on MLX5_CORE_EN + default y + help + Build support for NVMEoTCP acceleration in the NIC. + This includes Direct Data Placement and CRC offload. + Note: Support for hardware with this capability needs to be selected + for this option to become available. + config MLX5_SW_STEERING bool "Mellanox Technologies software-managed steering" depends on MLX5_CORE_EN && MLX5_ESWITCH
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -0
@@ -105,6 +105,8 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \ en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \ en_accel/ktls_tx.o en_accel/ktls_rx.o +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o + mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \ steering/dr_matcher.o steering/dr_rule.o \ steering/dr_icm_pool.o steering/dr_buddy.o \
drivers/net/ethernet/mellanox/mlx5/core/en.h +4 -0
@@ -319,6 +319,7 @@ struct mlx5e_params {int hard_mtu;bool ptp_rx;__be32 terminate_lkey_be; +bool nvmeotcp; };static inline u8 mlx5e_get_dcb_num_tc (struct mlx5e_params *params)
@@ -926,6 +927,9 @@ struct mlx5e_priv {#endif #ifdef CONFIG_MLX5_EN_TLS struct mlx5e_tls *tls; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct mlx5e_nvmeotcp *nvmeotcp;#endif struct devlink_health_reporter *tx_reporter;struct devlink_health_reporter *rx_reporter;
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +2 -2
@@ -77,7 +77,7 @@ enum { MLX5E_INNER_TTC_FT_LEVEL,MLX5E_FS_TT_UDP_FT_LEVEL =MLX5E_INNER_TTC_FT_LEVEL +1 ,MLX5E_FS_TT_ANY_FT_LEVEL =MLX5E_INNER_TTC_FT_LEVEL +1 , -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)MLX5E_ACCEL_FS_TCP_FT_LEVEL =MLX5E_INNER_TTC_FT_LEVEL +1 , #endif #ifdef CONFIG_MLX5_EN_ARFS
@@ -170,7 +170,7 @@ struct mlx5e_fs_any *mlx5e_fs_get_any(struct mlx5e_flow_steering *fs);void mlx5e_fs_set_any (struct mlx5e_flow_steering *fs,struct mlx5e_fs_any *any);struct mlx5e_fs_udp *mlx5e_fs_get_udp (struct mlx5e_flow_steering *fs);void mlx5e_fs_set_udp (struct mlx5e_flow_steering *fs,struct mlx5e_fs_udp *udp); -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS ) || defined(CONFIG_MLX5_EN_NVMEOTCP )struct mlx5e_accel_fs_tcp *mlx5e_fs_get_accel_tcp (struct mlx5e_flow_steering *fs);void mlx5e_fs_set_accel_tcp (struct mlx5e_flow_steering *fs,struct mlx5e_accel_fs_tcp *accel_tcp);#endif
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +8 -4
@@ -872,7 +872,8 @@ static void mlx5e_build_common_cq_param(struct mlx5_core_dev *mdev,void *cqc = param->cqc;MLX5_SET (cqc, cqc, uar_page, mdev->priv.uar->index); -if (MLX5_CAP_GEN (mdev, cqe_128_always) &&cache_line_size () >=128 ) +if (MLX5_CAP_GEN (mdev, cqe_128_always) && + (cache_line_size () >=128 || param->force_cqe128))MLX5_SET (cqc, cqc, cqe_sz,CQE_STRIDE_128_PAD ); }
@@ -902,6 +903,9 @@ static void mlx5e_build_rx_cq_param(struct mlx5_core_dev *mdev,void *cqc = param->cqc;u8 log_cq_size; +/* nvme-tcp offload mandates 128 byte cqes */ + param->force_cqe128 |=IS_ENABLED (CONFIG_MLX5_EN_NVMEOTCP ) && params->nvmeotcp; +switch (params->rq_wq_type) {case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ : hw_stridx =MLX5_CAP_GEN (mdev, mini_cqe_resp_stride_index);
@@ -1241,9 +1245,9 @@ static u8 mlx5e_build_async_icosq_log_wq_sz(struct mlx5_core_dev *mdev)return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE ; } -static void mlx5e_build_icosq_param (struct mlx5_core_dev *mdev, -u8 log_wq_size, -struct mlx5e_sq_param *param) +void mlx5e_build_icosq_param (struct mlx5_core_dev *mdev, +u8 log_wq_size, +struct mlx5e_sq_param *param) {void *sqc = param->sqc;void *wq =MLX5_ADDR_OF (sqc, sqc, wq);
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +3 -0
@@ -17,6 +17,7 @@ struct mlx5e_cq_param {struct mlx5_wq_param wq;u16 eq_ix;u8 cq_period_mode; +bool force_cqe128; };struct mlx5e_rq_param {
@@ -147,6 +148,8 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev,struct mlx5e_params *params, struct mlx5e_xsk_param *xsk, struct mlx5e_sq_param *param); +void mlx5e_build_icosq_param (struct mlx5_core_dev *mdev, +u8 log_wq_size,struct mlx5e_sq_param *param); int mlx5e_build_channel_param(struct mlx5_core_dev *mdev, struct mlx5e_params *params, u16 q_counter,
drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +3 -0
@@ -40,6 +40,7 @@#include "en_accel/ktls.h" #include "en_accel/ktls_txrx.h" #include <en_accel/macsec.h> +#include "en_accel/nvmeotcp.h" #include "en.h" #include "en/txrx.h"
@@ -202,11 +203,13 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,static inline int mlx5e_accel_init_rx (struct mlx5e_priv *priv) { +mlx5e_nvmeotcp_init_rx (priv);return mlx5e_ktls_init_rx (priv); }static inline void mlx5e_accel_cleanup_rx (struct mlx5e_priv *priv) { +mlx5e_nvmeotcp_cleanup_rx (priv);mlx5e_ktls_cleanup_rx (priv); }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h +1 -1
@@ -6,7 +6,7 @@#include "en/fs.h" -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS ) || defined(CONFIG_MLX5_EN_NVMEOTCP )int mlx5e_accel_fs_tcp_create (struct mlx5e_flow_steering *fs);void mlx5e_accel_fs_tcp_destroy (struct mlx5e_flow_steering *fs); struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_flow_steering *fs,
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c new file +216 -0
216 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h new file +121 -0
121 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +6 -0
@@ -38,6 +38,7 @@#include "en/ptp.h" #include "lib/clock.h" #include "en/fs_ethtool.h" +#include "en_accel/nvmeotcp.h" void mlx5e_ethtool_get_drvinfo (struct mlx5e_priv *priv,struct ethtool_drvinfo *drvinfo)
@@ -1929,6 +1930,11 @@ int mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool new_valreturn -EINVAL ; } +if (priv->channels.params.nvmeotcp) { +netdev_warn (priv->netdev,"Can't set CQE compression after ULP DDP NVMe-TCP offload\n" ); +return -EINVAL ; + } + new_params = priv->channels.params;MLX5E_SET_PFLAG (&new_params,MLX5E_PFLAG_RX_CQE_COMPRESS , new_val);if (rx_filter)
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c +2 -2
@@ -62,7 +62,7 @@ struct mlx5e_flow_steering {#ifdef CONFIG_MLX5_EN_ARFS struct mlx5e_arfs_tables *arfs;#endif -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS ) || defined(CONFIG_MLX5_EN_NVMEOTCP )struct mlx5e_accel_fs_tcp *accel_tcp;#endif struct mlx5e_fs_udp *udp;
@@ -1561,7 +1561,7 @@ void mlx5e_fs_set_any(struct mlx5e_flow_steering *fs, struct mlx5e_fs_any *any) fs->any = any; } -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS ) || defined(CONFIG_MLX5_EN_NVMEOTCP )struct mlx5e_accel_fs_tcp *mlx5e_fs_get_accel_tcp (struct mlx5e_flow_steering *fs) {return fs->accel_tcp;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +17 -0
@@ -50,6 +50,7 @@#include "en_accel/macsec.h" #include "en_accel/en_accel.h" #include "en_accel/ktls.h" +#include "en_accel/nvmeotcp.h" #include "lib/vxlan.h" #include "lib/clock.h" #include "en/port.h"
@@ -4184,6 +4185,13 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev, features &= ~NETIF_F_NETNS_LOCAL ; } +if (features & (NETIF_F_LRO |NETIF_F_GRO_HW )) { +if (params->nvmeotcp) { +netdev_warn (netdev,"Disabling HW-GRO/LRO, not supported after ULP DDP NVMe-TCP offload\n" ); + features &= ~(NETIF_F_LRO |NETIF_F_GRO_HW ); + } + } +mutex_unlock (&priv->state_lock);return features;
@@ -4940,6 +4948,9 @@ const struct net_device_ops mlx5e_netdev_ops = { .ndo_has_offload_stats = mlx5e_has_offload_stats, .ndo_get_offload_stats = mlx5e_get_offload_stats,#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + .ulp_ddp_ops = &mlx5e_nvmeotcp_ops, +#endif };static u32 mlx5e_choose_lro_timeout (struct mlx5_core_dev *mdev,u32 wanted_timeout)
@@ -5209,6 +5220,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)mlx5e_macsec_build_netdev (priv);mlx5e_ipsec_build_netdev (priv);mlx5e_ktls_build_netdev (priv); +mlx5e_nvmeotcp_build_netdev (priv); }void mlx5e_create_q_counters (struct mlx5e_priv *priv)
@@ -5279,6 +5291,10 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,if (err)mlx5_core_err (mdev,"TLS initialization failed, %d\n" , err); + err =mlx5e_nvmeotcp_init (priv); +if (err) +mlx5_core_err (mdev,"NVMEoTCP initialization failed, %d\n" , err); +mlx5e_health_create_reporters (priv);/* update XDP supported features */ mlx5e_set_xdp_feature (netdev);
@@ -5289,6 +5305,7 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,static void mlx5e_nic_cleanup (struct mlx5e_priv *priv) {mlx5e_health_destroy_reporters (priv); +mlx5e_nvmeotcp_cleanup (priv);mlx5e_ktls_cleanup (priv);mlx5e_fs_cleanup (priv->fs);debugfs_remove_recursive (priv->dfs_root);
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
@@ -1726,6 +1726,7 @@ static const int types[] = {MLX5_CAP_MACSEC ,MLX5_CAP_ADV_VIRTUALIZATION ,MLX5_CAP_CRYPTO , + MLX5_CAP_DEV_NVMEOTCP, };static void mlx5_hca_caps_free (struct mlx5_core_dev *dev)
Signature · 1 line
2.34.1
PATCH v12 20/26 net/mlx5e: TCP flow steering for nvme-tcp acceleration
In reply to Aurelien AptelFrom: Boris Pismenny <[email protected]>
Both nvme-tcp and tls acceleration require tcp flow steering. Add reference counter to share TCP flow steering structure.
--- .../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +11 -1
@@ -14,6 +14,7 @@ enum accel_fs_tcp_type {struct mlx5e_accel_fs_tcp {struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES ];struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES ]; +refcount_t user_count; };static enum mlx5_traffic_types fs_accel2tt (enum accel_fs_tcp_type i)
@@ -360,6 +361,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_flow_steering *fs)if (!accel_tcp)return ; +if (!refcount_dec_and_test (&accel_tcp->user_count)) +return ; +accel_fs_tcp_disable (fs);for (i =0 ; i <ACCEL_FS_TCP_NUM_TYPES ; i++)
@@ -371,12 +375,17 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_flow_steering *fs)int mlx5e_accel_fs_tcp_create (struct mlx5e_flow_steering *fs) { -struct mlx5e_accel_fs_tcp *accel_tcp; +struct mlx5e_accel_fs_tcp *accel_tcp =mlx5e_fs_get_accel_tcp (fs);int i, err;if (!MLX5_CAP_FLOWTABLE_NIC_RX (mlx5e_fs_get_mdev (fs), ft_field_support.outer_ip_version))return -EOPNOTSUPP ; +if (accel_tcp) { +refcount_inc (&accel_tcp->user_count); +return 0 ; + } + accel_tcp =kzalloc (sizeof (*accel_tcp),GFP_KERNEL );if (!accel_tcp)return -ENOMEM ;
@@ -392,6 +401,7 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_flow_steering *fs)if (err) goto err_destroy_tables; +refcount_set (&accel_tcp->user_count,1 );return 0 ; err_destroy_tables:
Signature · 1 line
2.34.1
PATCH v12 21/26 net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
NVMEoTCP offload uses buffer registration for ddp operation. Every request comprises from SG list that might consist from elements with multiple combination sizes, thus the appropriate way to perform buffer registration is with KLM UMRs.
UMR stands for user-mode memory registration, it is a mechanism to alter address translation properties of MKEY by posting WorkQueueElement aka WQE on send queue.
MKEY stands for memory key, MKEY are used to describe a region in memory that can be later used by HW.
KLM stands for {Key, Length, MemVa}, KLM_MKEY is indirect MKEY that enables to map multiple memory spaces with different sizes in unified MKEY. KLM UMR is a UMR that use to update a KLM_MKEY.
--- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 123 ++++++++++++++++++ .../mlx5/core/en_accel/nvmeotcp_utils.h | 25 ++++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 4 + 4 files changed, 155 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +3 -0
@@ -50,6 +50,9 @@ enum mlx5e_icosq_wqe_type { MLX5E_ICOSQ_WQE_SET_PSV_TLS, MLX5E_ICOSQ_WQE_GET_PSV_TLS,#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + MLX5E_ICOSQ_WQE_UMR_NVMEOTCP, +#endif };/* General */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +123 -0
141 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */ +#ifndef __MLX5E_NVMEOTCP_UTILS_H__ +#define __MLX5E_NVMEOTCP_UTILS_H__ + +#include "en.h" + +#define MLX5E_NVMEOTCP_FETCH_KLM_WQE (sq, pi) \ + ((struct mlx5e_umr_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_umr_wqe))) + +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS 0x4 + +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0 + +enum wqe_type { +KLM_UMR , +BSF_KLM_UMR , +SET_PSV_UMR , +BSF_UMR , +KLM_INV_UMR , +}; + +#endif /* __MLX5E_NVMEOTCP_UTILS_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +4 -0
@@ -1036,6 +1036,10 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)case MLX5E_ICOSQ_WQE_GET_PSV_TLS :mlx5e_ktls_handle_get_psv_completion (wi, sq);break ; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP : +break ;#endif default : netdev_WARN_ONCE(cq->netdev,
Signature · 1 line
2.34.1
PATCH v12 22/26 net/mlx5e: NVMEoTCP, queue init/teardown
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
Adds the ddp ops of sk_add, sk_del and offload limits.
When nvme-tcp establishes new queue/connection, the sk_add op is called. We allocate a hardware context to offload operations for this queue:
- use a steering rule based on the connection 5-tuple to mark packets of this queue/connection with a flow-tag in their completion (cqe)
- use a dedicated TIR to identify the queue and maintain the HW context
- use a dedicated ICOSQ to maintain the HW context by UMR postings
- use a dedicated tag buffer for buffer registration
- maintain static and progress HW contexts by posting the proper WQEs.
When nvme-tcp teardowns a queue/connection, the sk_del op is called. We teardown the queue and free the corresponding contexts.
The offload limits we advertise deal with the max SG supported.
[Re-enabled calling open/close icosq out of en_main.c]
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 + .../ethernet/mellanox/mlx5/core/en/rx_res.c | 28 + .../ethernet/mellanox/mlx5/core/en/rx_res.h | 4 + .../net/ethernet/mellanox/mlx5/core/en/tir.c | 15 + .../net/ethernet/mellanox/mlx5/core/en/tir.h | 2 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 563 +++++++++++++++++- .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 4 + .../mlx5/core/en_accel/nvmeotcp_utils.h | 41 ++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 8 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 15 +- 11 files changed, 680 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +4 -0
@@ -1026,6 +1026,10 @@ int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param);void mlx5e_destroy_rq (struct mlx5e_rq *rq);struct mlx5e_sq_param ; +int mlx5e_open_icosq (struct mlx5e_channel *c,struct mlx5e_params *params, +struct mlx5e_sq_param *param,struct mlx5e_icosq *sq, +work_func_t recover_work_func); +void mlx5e_close_icosq (struct mlx5e_icosq *sq);int mlx5e_open_xdpsq (struct mlx5e_channel *c,struct mlx5e_params *params,struct mlx5e_sq_param *param,struct xsk_buff_pool *xsk_pool,struct mlx5e_xdpsq *sq,bool is_redirect);
drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c +28 -0
@@ -611,6 +611,34 @@ struct mlx5e_rss_params_hash mlx5e_rx_res_get_current_hash(struct mlx5e_rx_res *return mlx5e_rss_get_hash (res->rss[0 ]); } +int mlx5e_rx_res_nvmeotcp_tir_create (struct mlx5e_rx_res *res,unsigned int rxq,bool crc_rx, +u32 tag_buf_id,struct mlx5e_tir *tir) +{ +bool inner_ft_support = res->features &MLX5E_RX_RES_FEATURE_INNER_FT ; +struct mlx5e_tir_builder *builder; +u32 rqtn; +int err; + + builder =mlx5e_tir_builder_alloc (false); +if (!builder) +return -ENOMEM ; + + rqtn =mlx5e_rx_res_get_rqtn_direct (res, rxq); + +mlx5e_tir_builder_build_rqt (builder, res->mdev->mlx5e_res.hw_objs.td.tdn, rqtn, + inner_ft_support); +mlx5e_tir_builder_build_direct (builder); +mlx5e_tir_builder_build_nvmeotcp (builder, crc_rx, tag_buf_id); +down_read (&res->pkt_merge_param_sem); +mlx5e_tir_builder_build_packet_merge (builder, &res->pkt_merge_param); + err =mlx5e_tir_init (tir, builder, res->mdev, false); +up_read (&res->pkt_merge_param_sem); + +mlx5e_tir_builder_free (builder); + +return err; +} +int mlx5e_rx_res_tls_tir_create (struct mlx5e_rx_res *res,unsigned int rxq,struct mlx5e_tir *tir) {
drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h +4 -0
@@ -66,4 +66,8 @@ struct mlx5e_rss_params_hash mlx5e_rx_res_get_current_hash(struct mlx5e_rx_res */* Accel TIRs */ int mlx5e_rx_res_tls_tir_create (struct mlx5e_rx_res *res,unsigned int rxq,struct mlx5e_tir *tir); + +int mlx5e_rx_res_nvmeotcp_tir_create (struct mlx5e_rx_res *res,unsigned int rxq,bool crc_rx, +u32 tag_buf_id,struct mlx5e_tir *tir); +#endif /* __MLX5_EN_RX_RES_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en/tir.c +15 -0
@@ -143,6 +143,21 @@ void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder)MLX5_SET (tirc, tirc, rx_hash_fn,MLX5_RX_HASH_FN_INVERTED_XOR8 ); } +void mlx5e_tir_builder_build_nvmeotcp (struct mlx5e_tir_builder *builder,bool crc_rx, +u32 tag_buf_id) +{ +void *tirc =mlx5e_tir_builder_get_tirc (builder); + +WARN_ON (builder->modify); + +MLX5_SET (tirc, tirc, nvmeotcp_zero_copy_en,1 ); +MLX5_SET (tirc, tirc, nvmeotcp_tag_buffer_table_id, tag_buf_id); +MLX5_SET (tirc, tirc, nvmeotcp_crc_en, !!crc_rx); +MLX5_SET (tirc, tirc, self_lb_block, +MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST | +MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST ); +} +void mlx5e_tir_builder_build_tls (struct mlx5e_tir_builder *builder) {void *tirc =mlx5e_tir_builder_get_tirc (builder);
drivers/net/ethernet/mellanox/mlx5/core/en/tir.h +2 -0
@@ -35,6 +35,8 @@ void mlx5e_tir_builder_build_rss(struct mlx5e_tir_builder *builder, bool inner);void mlx5e_tir_builder_build_direct (struct mlx5e_tir_builder *builder);void mlx5e_tir_builder_build_tls (struct mlx5e_tir_builder *builder); +void mlx5e_tir_builder_build_nvmeotcp (struct mlx5e_tir_builder *builder,bool crc_rx, +u32 tag_buf_id);struct mlx5_core_dev ;
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +6 -0
@@ -52,6 +52,7 @@ enum mlx5e_icosq_wqe_type {#endif #ifdef CONFIG_MLX5_EN_NVMEOTCP MLX5E_ICOSQ_WQE_UMR_NVMEOTCP, + MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP,#endif };
@@ -224,6 +225,11 @@ struct mlx5e_icosq_wqe_info {struct {struct mlx5e_ktls_rx_resync_buf *buf; } tls_get_params; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +struct { +struct mlx5e_nvmeotcp_queue *queue; + } nvmeotcp_q;#endif }; };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +559 -4
639 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +4 -0
@@ -106,6 +106,10 @@ void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv);int mlx5e_nvmeotcp_init (struct mlx5e_priv *priv);int set_ulp_ddp_nvme_tcp (struct net_device *netdev,bool enable);void mlx5e_nvmeotcp_cleanup (struct mlx5e_priv *priv); +struct mlx5e_nvmeotcp_queue * +mlx5e_nvmeotcp_get_queue (struct mlx5e_nvmeotcp *nvmeotcp,int id); +void mlx5e_nvmeotcp_put_queue (struct mlx5e_nvmeotcp_queue *queue); +void mlx5e_nvmeotcp_ctx_complete (struct mlx5e_icosq_wqe_info *wi);static inline void mlx5e_nvmeotcp_init_rx (struct mlx5e_priv *priv) {}void mlx5e_nvmeotcp_cleanup_rx (struct mlx5e_priv *priv);extern const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops;
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h +41 -0
57 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -4
@@ -1774,9 +1774,9 @@ void mlx5e_tx_err_cqe_work(struct work_struct *recover_work)mlx5e_reporter_tx_err_cqe (sq); } -static int mlx5e_open_icosq (struct mlx5e_channel *c,struct mlx5e_params *params, -struct mlx5e_sq_param *param,struct mlx5e_icosq *sq, -work_func_t recover_work_func) +int mlx5e_open_icosq (struct mlx5e_channel *c,struct mlx5e_params *params, +struct mlx5e_sq_param *param,struct mlx5e_icosq *sq, +work_func_t recover_work_func) {struct mlx5e_create_sq_param csp = {};int err;
@@ -1820,7 +1820,7 @@ void mlx5e_deactivate_icosq(struct mlx5e_icosq *icosq)synchronize_net ();/* Sync with NAPI. */ } -static void mlx5e_close_icosq (struct mlx5e_icosq *sq) +void mlx5e_close_icosq (struct mlx5e_icosq *sq) {if (sq->ktls_resync)mlx5e_ktls_rx_resync_destroy_resp_list (sq->ktls_resync);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +13 -2
@@ -53,6 +53,7 @@#include "en_accel/macsec.h" #include "en_accel/ipsec_rxtx.h" #include "en_accel/ktls_txrx.h" +#include "en_accel/nvmeotcp.h" #include "en/xdp.h" #include "en/xsk/rx.h" #include "en/health.h"
@@ -932,16 +933,23 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq) ci =mlx5_wq_cyc_ctr2ix (&sq->wq, sqcc); wi = &sq->db.wqe_info[ci]; sqcc += wi->num_wqebbs; -#ifdef CONFIG_MLX5_EN_TLS switch (wi->wqe_type) { +#ifdef CONFIG_MLX5_EN_TLS case MLX5E_ICOSQ_WQE_SET_PSV_TLS :mlx5e_ktls_handle_ctx_completion (wi);break ;case MLX5E_ICOSQ_WQE_GET_PSV_TLS :mlx5e_ktls_handle_get_psv_completion (wi, sq);break ; - }#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP : +mlx5e_nvmeotcp_ctx_complete (wi); +break ; +#endif +default : +break ; + } } sq->cc = sqcc; }
@@ -1040,6 +1048,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)#ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP :break ; +case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP : +mlx5e_nvmeotcp_ctx_complete (wi); +break ;#endif default : netdev_WARN_ONCE(cq->netdev,
Signature · 1 line
2.34.1
PATCH v12 23/26 net/mlx5e: NVMEoTCP, ddp setup and resync
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
NVMEoTCP offload uses buffer registration for every NVME request to perform direct data placement. This is achieved by creating a NIC HW mapping between the CCID (command capsule ID) to the set of buffers that compose the request. The registration is implemented via MKEY for which we do fast/async mapping using KLM UMR WQE.
The buffer registration takes place when the ULP calls the ddp_setup op which is done before they send their corresponding request to the other side (e.g nvmf target). We don't wait for the completion of the registration before returning back to the ulp. The reason being that the HW mapping should be in place fast enough vs the RTT it would take for the request to be responded. If this doesn't happen, some IO may not be ddp-offloaded, but that doesn't stop the overall offloading session.
When the offloading HW gets out of sync with the protocol session, a hardware/software handshake takes place to resync. The ddp_resync op is the part of the handshake where the SW confirms to the HW that a indeed they identified correctly a PDU header at a certain TCP sequence number. This allows the HW to resume the offload.
The 1st part of the handshake is when the HW identifies such sequence number in an arriving packet. A special mark is made on the completion (cqe) and then the mlx5 driver invokes the ddp resync_request callback advertised by the ULP in the ddp context - this is in downstream patch.
--- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 146 +++++++++++++++++- 1 file changed, 144 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +144 -2
@@ -683,19 +683,156 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,mlx5e_nvmeotcp_put_queue (queue); } +static bool +mlx5e_nvmeotcp_validate_small_sgl_suffix (struct scatterlist *sg,int sg_len,int mtu) +{ +int i, hole_size, hole_len, chunk_size =0 ; + +for (i =1 ; i < sg_len; i++) + chunk_size +=sg_dma_len (&sg[i]); + +if (chunk_size >= mtu) +return true; + + hole_size = mtu - chunk_size -1 ; + hole_len =DIV_ROUND_UP (hole_size,PAGE_SIZE ); + +if (sg_len + hole_len >MAX_SKB_FRAGS ) +return false; + +return true; +} + +static bool +mlx5e_nvmeotcp_validate_big_sgl_suffix (struct scatterlist *sg,int sg_len,int mtu) +{ +int i, j, last_elem, window_idx, window_size =MAX_SKB_FRAGS -1 ; +int chunk_size =0 ; + + last_elem = sg_len - window_size; + window_idx = window_size; + +for (j =1 ; j < window_size; j++) + chunk_size +=sg_dma_len (&sg[j]); + +for (i =1 ; i <= last_elem; i++, window_idx++) { + chunk_size +=sg_dma_len (&sg[window_idx]); +if (chunk_size < mtu -1 ) +return false; + + chunk_size -=sg_dma_len (&sg[i]); + } + +return true; +} + +/* This function makes sure that the middle/suffix of a PDU SGL meets the +* restriction of MAX_SKB_FRAGS. There are two cases here: +* 1. sg_len < MAX_SKB_FRAGS - the extreme case here is a packet that consists +* of one byte from the first SG element + the rest of the SGL and the remaining +* space of the packet will be scattered to the WQE and will be pointed by +* SKB frags. +* 2. sg_len => MAX_SKB_FRAGS - the extreme case here is a packet that consists +* of one byte from middle SG element + 15 continuous SG elements + one byte +* from a sequential SG element or the rest of the packet. +*/ +static bool +mlx5e_nvmeotcp_validate_sgl_suffix (struct scatterlist *sg,int sg_len,int mtu) +{ +int ret; + +if (sg_len <MAX_SKB_FRAGS ) + ret =mlx5e_nvmeotcp_validate_small_sgl_suffix (sg, sg_len, mtu); +else + ret =mlx5e_nvmeotcp_validate_big_sgl_suffix (sg, sg_len, mtu); + +return ret; +} + +static bool +mlx5e_nvmeotcp_validate_sgl_prefix (struct scatterlist *sg,int sg_len,int mtu) +{ +int i, hole_size, hole_len, tmp_len, chunk_size =0 ; + + tmp_len =min_t (int, sg_len,MAX_SKB_FRAGS ); + +for (i =0 ; i < tmp_len; i++) + chunk_size +=sg_dma_len (&sg[i]); + +if (chunk_size >= mtu) +return true; + + hole_size = mtu - chunk_size; + hole_len =DIV_ROUND_UP (hole_size,PAGE_SIZE ); + +if (tmp_len + hole_len >MAX_SKB_FRAGS ) +return false; + +return true; +} + +/* This function is responsible to ensure that a PDU could be offloaded. +* PDU is offloaded by building a non-linear SKB such that each SGL element is +* placed in frag, thus this function should ensure that all packets that +* represent part of the PDU won't exaggerate from MAX_SKB_FRAGS SGL. +* In addition NVMEoTCP offload has one PDU offload for packet restriction. +* Packet could start with a new PDU and then we should check that the prefix +* of the PDU meets the requirement or a packet can start in the middle of SG +* element and then we should check that the suffix of PDU meets the requirement. +*/ +static bool +mlx5e_nvmeotcp_validate_sgl (struct scatterlist *sg,int sg_len,int mtu) +{ +int max_hole_frags; + + max_hole_frags =DIV_ROUND_UP (mtu,PAGE_SIZE ); +if (sg_len + max_hole_frags <=MAX_SKB_FRAGS ) +return true; + +if (!mlx5e_nvmeotcp_validate_sgl_prefix (sg, sg_len, mtu) || + !mlx5e_nvmeotcp_validate_sgl_suffix (sg, sg_len, mtu)) +return false; + +return true; +} +static int mlx5e_nvmeotcp_ddp_setup (struct net_device *netdev,struct sock *sk,struct ulp_ddp_io *ddp) { +struct scatterlist *sg = ddp->sg_table.sgl; +struct mlx5e_nvmeotcp_queue_entry *nvqt;struct mlx5e_nvmeotcp_queue *queue; +struct mlx5_core_dev *mdev; +int i, size =0 , count =0 ; queue =container_of (ulp_ddp_get_ctx (sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + mdev = queue->priv->mdev; + count =dma_map_sg (mdev->device, ddp->sg_table.sgl, ddp->nents, +DMA_FROM_DEVICE ); + +if (count <=0 ) +return -EINVAL ; -/* Placeholder - map_sg and initializing the count */ +if (WARN_ON (count >mlx5e_get_max_sgl (mdev))) +return -ENOSPC ; + +if (!mlx5e_nvmeotcp_validate_sgl (sg, count,READ_ONCE (netdev->mtu))) +return -EOPNOTSUPP ; + +for (i =0 ; i < count; i++) + size +=sg_dma_len (&sg[i]); + + nvqt = &queue->ccid_table[ddp->command_id]; + nvqt->size = size; + nvqt->ddp = ddp; + nvqt->sgl = sg; + nvqt->ccid_gen++; + nvqt->sgl_length = count; +mlx5e_nvmeotcp_post_klm_wqe (queue,KLM_UMR , ddp->command_id, count); -mlx5e_nvmeotcp_post_klm_wqe (queue,KLM_UMR , ddp->command_id,0 );return 0 ; }
@@ -718,6 +855,11 @@ static void mlx5e_nvmeotcp_ddp_resync(struct net_device *netdev, struct sock *sk, u32 seq) { + struct mlx5e_nvmeotcp_queue *queue = + container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + + queue->after_resync_cqe =1 ; +mlx5e_nvmeotcp_rx_post_static_params_wqe (queue, seq); } struct mlx5e_nvmeotcp_queue *
Signature · 1 line
2.34.1
PATCH v12 24/26 net/mlx5e: NVMEoTCP, async ddp invalidation
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
After the ULP consumed the buffers of the offloaded request, it calls the ddp_teardown op to release the NIC mapping for them and allow the NIC to reuse the HW contexts associated with offloading this IO. We do a fast/async un-mapping via UMR WQE. In this case, the ULP does holds off with completing the request towards the upper/application layers until the HW unmapping is done.
When the corresponding CQE is received, a notification is done via the the teardown_done ddp callback advertised by the ULP in the ddp context.
--- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++ .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 66 ++++++++++++++++--- .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 1 + .../net/ethernet/mellanox/mlx5/core/en_rx.c | 6 ++ 4 files changed, 67 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +4 -0
@@ -52,6 +52,7 @@ enum mlx5e_icosq_wqe_type {#endif #ifdef CONFIG_MLX5_EN_NVMEOTCP MLX5E_ICOSQ_WQE_UMR_NVMEOTCP , +MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE , MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP, #endif };
@@ -230,6 +231,9 @@ struct mlx5e_icosq_wqe_info {struct {struct mlx5e_nvmeotcp_queue *queue; } nvmeotcp_q; +struct { +struct mlx5e_nvmeotcp_queue_entry *entry; + } nvmeotcp_qe;#endif }; };
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +56 -10
126 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +1 -0
@@ -109,6 +109,7 @@ void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);struct mlx5e_nvmeotcp_queue *mlx5e_nvmeotcp_get_queue (struct mlx5e_nvmeotcp *nvmeotcp,int id);void mlx5e_nvmeotcp_put_queue (struct mlx5e_nvmeotcp_queue *queue); +void mlx5e_nvmeotcp_ddp_inv_done (struct mlx5e_icosq_wqe_info *wi);void mlx5e_nvmeotcp_ctx_complete (struct mlx5e_icosq_wqe_info *wi);static inline void mlx5e_nvmeotcp_init_rx (struct mlx5e_priv *priv) {}void mlx5e_nvmeotcp_cleanup_rx (struct mlx5e_priv *priv);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +6 -0
@@ -943,6 +943,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)break ;#endif #ifdef CONFIG_MLX5_EN_NVMEOTCP +case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE : +mlx5e_nvmeotcp_ddp_inv_done (wi); +break ;case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP :mlx5e_nvmeotcp_ctx_complete (wi);break ;
@@ -1048,6 +1051,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)#ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP :break ; +case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE : +mlx5e_nvmeotcp_ddp_inv_done (wi); +break ;case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP :mlx5e_nvmeotcp_ctx_complete (wi);break ;
Signature · 1 line
2.34.1
PATCH v12 25/26 net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload
In reply to Aurelien AptelFrom: Ben Ben-Ishay <[email protected]>
This patch implements the data-path for direct data placement (DDP) and DDGST offloads. NVMEoTCP DDP constructs an SKB from each CQE, while pointing at NVME destination buffers. In turn, this enables the offload, as the NVMe-TCP layer will skip the copy when src == dst.
Additionally, this patch adds support for DDGST (CRC32) offload. HW will report DDGST offload only if it has not encountered an error in the received packet. We pass this indication in skb->ulp_crc up the stack to NVMe-TCP to skip computing the DDGST if all corresponding SKBs were verified by HW.
This patch also handles context resynchronization requests made by NIC HW. The resync request is passed to the NVMe-TCP layer to be handled at a later point in time.
Finally, we also use the skb->ulp_ddp bit to avoid skb_condense. This is critical as every SKB that uses DDP has a hole that fits perfectly with skb_condense's policy, but filling this hole is counter-productive as the data there already resides in its destination buffer.
This work has been done on pre-silicon functional simulator, and hence data-path performance numbers are not provided.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 + .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 345 ++++++++++++++++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.h | 37 ++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 44 ++- 5 files changed, 419 insertions(+), 15 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -105,7 +105,7 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \ en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \ en_accel/ktls_tx.o en_accel/ktls_rx.o -mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \ steering/dr_matcher.o steering/dr_rule.o \
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +6 -0
@@ -526,4 +526,10 @@ static inline struct mlx5e_mpw_info *mlx5e_get_mpw_info(struct mlx5e_rq *rq, int return (struct mlx5e_mpw_info *)((char *)rq->mpwqe.info + array_size(i, isz)); } + +static inline struct mlx5e_wqe_frag_info *get_frag (struct mlx5e_rq *rq,u16 ix) +{ +return &rq->wqe.frags[ix << rq->wqe.info.log_num_frags]; +} +#endif
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c new file +345 -0
345 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */ +#ifndef __MLX5E_NVMEOTCP_RXTX_H__ +#define __MLX5E_NVMEOTCP_RXTX_H__ + +#ifdef CONFIG_MLX5_EN_NVMEOTCP + +#include <linux/skbuff.h> +#include "en_accel/nvmeotcp.h" + +bool +mlx5e_nvmeotcp_rebuild_rx_skb (struct mlx5e_rq *rq,struct sk_buff *skb, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt); + +static inline int mlx5_nvmeotcp_get_headlen (struct mlx5_cqe64 *cqe,u32 cqe_bcnt) +{ +struct mlx5e_cqe128 *cqe128; + +if (!cqe_is_nvmeotcp_zc (cqe)) +return cqe_bcnt; + + cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64); +return be16_to_cpu (cqe128->hlen); +} + +#else + +static inline bool +mlx5e_nvmeotcp_rebuild_rx_skb (struct mlx5e_rq *rq,struct sk_buff *skb, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt) +{return true; } + +static inline int mlx5_nvmeotcp_get_headlen (struct mlx5_cqe64 *cqe,u32 cqe_bcnt) +{return cqe_bcnt; } + +#endif /* CONFIG_MLX5_EN_NVMEOTCP */ +#endif /* __MLX5E_NVMEOTCP_RXTX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +30 -14
128 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v12 26/26 net/mlx5e: NVMEoTCP, statistics
In reply to Aurelien AptelNVMEoTCP offload statistics include both control and data path
statistic: counters for the netdev ddp ops, offloaded packets/bytes, resync and dropped packets.
Expose the statistics using ulp_ddp_ops->get_stats() instead of the regular statistics flow.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 3 +- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 53 ++++++++++++--- .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 16 +++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 11 +++- .../mlx5/core/en_accel/nvmeotcp_stats.c | 66 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/en_stats.h | 7 ++ 6 files changed, 145 insertions(+), 11 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -1
@@ -105,7 +105,8 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \ en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \ en_accel/ktls_tx.o en_accel/ktls_rx.o -mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o \ + en_accel/nvmeotcp_rxtx.o en_accel/nvmeotcp_stats.o mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \ steering/dr_matcher.o steering/dr_rule.o \
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +44 -9
139 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +16 -0
@@ -9,6 +9,15 @@#include "en.h" #include "en/params.h" +struct mlx5e_nvmeotcp_sw_stats { +atomic64_t rx_nvmeotcp_sk_add; +atomic64_t rx_nvmeotcp_sk_add_fail; +atomic64_t rx_nvmeotcp_sk_del; +atomic64_t rx_nvmeotcp_ddp_setup; +atomic64_t rx_nvmeotcp_ddp_setup_fail; +atomic64_t rx_nvmeotcp_ddp_teardown; +}; +struct mlx5e_nvmeotcp_queue_entry {struct mlx5e_nvmeotcp_queue *queue;u32 sgl_length;
@@ -52,6 +61,7 @@ struct mlx5e_nvmeotcp_queue_handler {* @sk: The socket used by the NVMe-TCP queue * @crc_rx: CRC Rx offload indication for this queue * @priv: mlx5e netdev priv +* @sw_stats: Global software statistics for nvmeotcp offload * @static_params_done: Async completion structure for the initial umr mapping * synchronization * @sq_lock: Spin lock for the icosq
@@ -88,6 +98,7 @@ struct mlx5e_nvmeotcp_queue { u8 crc_rx:1;/* for ddp invalidate flow */ struct mlx5e_priv *priv; +struct mlx5e_nvmeotcp_sw_stats *sw_stats;/* end of data-path section */ struct completion static_params_done;
@@ -97,6 +108,7 @@ struct mlx5e_nvmeotcp_queue { };struct mlx5e_nvmeotcp { +struct mlx5e_nvmeotcp_sw_stats sw_stats;struct ida queue_ids;struct rhashtable queue_hash;bool enabled;
@@ -113,6 +125,7 @@ void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi);void mlx5e_nvmeotcp_ctx_complete (struct mlx5e_icosq_wqe_info *wi);static inline void mlx5e_nvmeotcp_init_rx (struct mlx5e_priv *priv) {}void mlx5e_nvmeotcp_cleanup_rx (struct mlx5e_priv *priv); +int mlx5e_nvmeotcp_get_stats (struct mlx5e_priv *priv,struct ethtool_ulp_ddp_stats *stats);extern const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops;#else
@@ -122,5 +135,8 @@ static inline void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) {}static inline int set_ulp_ddp_nvme_tcp (struct net_device *dev,bool en) {return -EOPNOTSUPP ; }static inline void mlx5e_nvmeotcp_init_rx (struct mlx5e_priv *priv) {}static inline void mlx5e_nvmeotcp_cleanup_rx (struct mlx5e_priv *priv) {} +static inline int mlx5e_nvmeotcp_get_stats (struct mlx5e_priv *priv, +struct ethtool_ulp_ddp_stats *stats) +{return 0 ; }#endif #endif /* __MLX5E_NVMEOTCP_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c +10 -1
@@ -140,6 +140,7 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skbint ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy =0 ;struct net_device *netdev = rq->netdev;struct mlx5e_priv *priv =netdev_priv (netdev); +struct mlx5e_rq_stats *stats = rq->stats;struct mlx5e_nvmeotcp_queue_entry *nqe;skb_frag_t org_frags[MAX_SKB_FRAGS ];struct mlx5e_nvmeotcp_queue *queue;
@@ -151,12 +152,14 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb queue =mlx5e_nvmeotcp_get_queue (priv->nvmeotcp, queue_id);if (unlikely (!queue)) {dev_kfree_skb_any (skb); + stats->nvmeotcp_drop++;return false; } cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);if (cqe_is_nvmeotcp_resync (cqe)) {nvmeotcp_update_resync (queue, cqe128); + stats->nvmeotcp_resync++;mlx5e_nvmeotcp_put_queue (queue);return true; }
@@ -230,7 +233,8 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb org_nr_frags, frag_index); } - + stats->nvmeotcp_packets++; + stats->nvmeotcp_bytes += cclen;mlx5e_nvmeotcp_put_queue (queue);return true; }
@@ -242,6 +246,7 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb,int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy =0 ;struct net_device *netdev = rq->netdev;struct mlx5e_priv *priv =netdev_priv (netdev); +struct mlx5e_rq_stats *stats = rq->stats;struct mlx5e_nvmeotcp_queue_entry *nqe;struct mlx5e_nvmeotcp_queue *queue;struct mlx5e_cqe128 *cqe128;
@@ -251,12 +256,14 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb, queue =mlx5e_nvmeotcp_get_queue (priv->nvmeotcp, queue_id);if (unlikely (!queue)) {dev_kfree_skb_any (skb); + stats->nvmeotcp_drop++;return false; } cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);if (cqe_is_nvmeotcp_resync (cqe)) {nvmeotcp_update_resync (queue, cqe128); + stats->nvmeotcp_resync++;mlx5e_nvmeotcp_put_queue (queue);return true; }
@@ -330,6 +337,8 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb, hlen + cclen, remaining); } + stats->nvmeotcp_packets++; + stats->nvmeotcp_bytes += cclen;mlx5e_nvmeotcp_put_queue (queue);return true; }
drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c new file +66 -0
66 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +7 -0
@@ -128,6 +128,7 @@ void mlx5e_stats_rmon_get(struct mlx5e_priv *priv, const struct ethtool_rmon_hist_range **ranges);void mlx5e_get_link_ext_stats (struct net_device *dev,struct ethtool_link_ext_stats *stats); +void mlx5e_stats_ulp_ddp_get (struct mlx5e_priv *priv,struct ethtool_ulp_ddp_stats *stats);/* Concrete NIC Stats */
@@ -385,6 +386,12 @@ struct mlx5e_rq_stats {u64 tls_resync_res_skip;u64 tls_err;#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP +u64 nvmeotcp_drop; +u64 nvmeotcp_resync; +u64 nvmeotcp_packets; +u64 nvmeotcp_bytes; +#endif };struct mlx5e_sq_stats {
Signature · 1 line
2.34.1