RDMA over Fibre Channel
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 RDMA over Fibre Channel
Quoted 5 lines · On Wed, Apr 18, 2018@02:42:40AM -0700, [email protected] wrote:
Eventhough it is inspired from the Soft RoCE driver, the underlying transport layer is FC-NVMe (short for 'NVMe over fibre channel'). The request, response and completion state machines in the driver have been heavily modified to adapt to the Exchange based Data transfer mechanism of Fibre channel.
That sounds like a bad joke. Please stop abusing the NVMe code for this otherwise reasonable idea. You should be able to layer this over plain FCP just fine.
PATCH 1/1 RDMA over Fibre Channel
Hi Christoph,
The current implementation of RDMA over Fibre channel uses NVMe for the below reasons:
1. Existing FC-NVMe HBA and FC network can be used without requiring any changes.
2. NVMe namespace based discovery is used for RDMA node discovery.
3. FC-NVMe provides us a way to achieve Zero copy TX/RX for non-block workloads.
Although we concur with the idea of RDMA directly over Fibre channel, the actual implementation addressing the above reasons requires standardization and coordination with FC HBA vendors and other SAN ecosystem players. This effort is ongoing within our organization (Brocade
at Broadcom). However, there is a business case for the current soft
RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides existing Fibre channel customers a way to utilize existing FC network to transport RDMA workloads as well. While doing this we are making sure NVMe block traffic also can happen on the same FC network.
The below link gives more technical details. We are glad to discuss any further details on the same.
https://github.com/brocade/RDMAoverFC/blob/master/RDMA%20over%20FC.pdf
Regards,
Muneendra, Amit & Anand.
-----Original Message-----
From: Christoph Hellwig [mailto:[email protected]]
Sent: Wednesday, April 18, 2018 3:52 PM
To: muneendra.kumar at broadcom.com
Cc: linux-rdma at vger.kernel.org; amit.tyagi at broadcom.com; anand.sundaram at broadcom.com; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel
Quoted 5 lines · muneendra.kumar at broadcom.com
Eventhough it is inspired from the Soft RoCE driver, the underlying transport layer is FC-NVMe (short for 'NVMe over fibre channel'). The request, response and completion state machines in the driver have been heavily modified to adapt to the Exchange based Data transfer mechanism of Fibre channel.
That sounds like a bad joke. Please stop abusing the NVMe code for this otherwise reasonable idea. You should be able to layer this over plain FCP just fine.
PATCH 1/1 RDMA over Fibre Channel
Quoted 9 lines · Muneendra Kumar M
Although we concur with the idea of RDMA directly over Fibre channel, the actual implementation addressing the above reasons requires standardization and coordination with FC HBA vendors and other SAN ecosystem players. This effort is ongoing within our organization (Brocade
at Broadcom). However, there is a business case for the current soft
RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides existing Fibre channel customers a way to utilize existing FC network to transport RDMA workloads as well. While doing this we are making sure NVMe block traffic also can happen on the same FC network.
There might be a business case for you, but with my Linux NVMe (co-)maintainer hat on I'll have to tell you that this abuse of the Linux nvme code is a complete no-go.
And even if it wasn't we'd still require the protocol be ratified by the NVMe technical working group first.
PATCH 1/1 RDMA over Fibre Channel
Hi Christoph,
Just wanted to understand more on your concerns on the mods done to Linux NVMe.
The whole work was to tunnel IB protocol over existing NVMe protocol. To do this we first made sure NVMe stack (host, target) is able to send block traffic and non-block (object based ) traffic. To do this, no changes were required in the NVMe protocol itself. Only the target stack needed some modifications to vector
(a) NVMe block traffic to backend NVMe Namespace block driver
(b) non-block IB protocol traffic to RFC transport layer
The NVMe changes are restricted to below:
drivers/nvme/target/fc.c | 94 +- drivers/nvme/target/io-cmd.c | 44 +- include/linux/nvme-fc-driver.h | 6 +
Of the above only the io-cmd.c is generic target component. The rest are contained under FC stack.
Finally I respect your decision. The above intent is to demonstrate existing NVMe protocol can support both block and non-block workloads. If this requires working with NVMe core working group, kindly let us know how to accomplish this.
Thanks,
-anand
-----Original Message-----
From: Christoph Hellwig [mailto:[email protected]]
Sent: Wednesday, April 18, 2018 6:49 PM
To: Muneendra Kumar M <muneendra.kumar at broadcom.com>
Cc: Christoph Hellwig <hch at infradead.org>; linux-rdma at vger.kernel.org; Amit Kumar Tyagi <amit.tyagi at broadcom.com>; Anand Nataraja Sundaram <anand.sundaram at broadcom.com>; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel
Quoted 4 lines · Muneendra Kumar M
Although we concur with the idea of RDMA directly over Fibre channel, the actual implementation addressing the above reasons requires standardization and coordination with FC HBA vendors and other SAN ecosystem players. This effort is ongoing within our organization
(Brocade
Quoted 5 lines
at Broadcom). However, there is a business case for the current soft
RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides existing Fibre channel customers a way to utilize existing FC network to transport RDMA workloads as well. While doing this we are making sure NVMe block traffic also can happen on the same FC network.
There might be a business case for you, but with my Linux NVMe (co-)maintainer hat on I'll have to tell you that this abuse of the Linux nvme code is a complete no-go.
And even if it wasn't we'd still require the protocol be ratified by the NVMe technical working group first.
PATCH 1/1 RDMA over Fibre Channel
Quoted 11 lines · Anand Nataraja Sundaram
Just wanted to understand more on your concerns on the mods done to Linux NVMe.
The whole work was to tunnel IB protocol over existing NVMe protocol. To do this we first made sure NVMe stack (host, target) is able to send block traffic and non-block (object based ) traffic. To do this, no changes were required in the NVMe protocol itself. Only the target stack needed some modifications to vector
(a) NVMe block traffic to backend NVMe Namespace block driver
(b) non-block IB protocol traffic to RFC transport layer
Quoted 4 lines
The NVMe changes are restricted to below:
drivers/nvme/target/fc.c | 94 +- drivers/nvme/target/io-cmd.c | 44 +- include/linux/nvme-fc-driver.h | 6 +
You forgot the larger chunks of Linux NVMe code you copied while stripping the copyrights and incorrectly relicensing it to a BSD-like license.
The point is that IFF you really want to do RDMA over NVMe you need to defined a new NVMe I/O command set for it and get it standardized. If that is done we could do a proper upper level protocol interface for it, instead of just hacking it into the protocol and code through the backdoor. But as said before there is no upside of using NVMe, I can see the interest in layering on top of FCP to reuse existing hardware accelerations, similar to how NVMe layers on top of FCP for that reason, but there isn't really any value in throwing in another NVMe layer.
PATCH 1/1 RDMA over Fibre Channel
Agreed some of the host nvme code was wrongly duplicated under BSD-like license indrivers/infiniband/sw/rfc/rfc_tb.c | 795 +++++++++++++
Bottom line: We need both NVMe host and NVMe target stack changes to tunnel RDMA over FC-NVMe.
This exercise just proved that RDMA can be tunneled over FC-NVMe. I agree we need some standardization to transport RDMA workload over FC networks.
The clear advantage of doing RDMA over NVMe is that we could do end to end zero-copy between RDMA applications. Doing RDMA over SCSI-FCP incurs a one-copy penalty between RDMA applications.
However doing RDMA over FC directly (as a new FC-4 Upper Level Protocol type) is also a possibility. Here FC-VI could also be considered. All this required using new HBAs.
As a FC-SAN Community, we will work out which is the best route for RDMA over FC standardization.
Thanks for your feedback,
-anand
-----Original Message-----
From: Christoph Hellwig [mailto:[email protected]]
Sent: Thursday, April 19, 2018 3:10 PM
To: Anand Nataraja Sundaram <anand.sundaram at broadcom.com>
Cc: Christoph Hellwig <hch at infradead.org>; Muneendra Kumar M <muneendra.kumar at broadcom.com>; linux-rdma at vger.kernel.org; Amit Kumar Tyagi <amit.tyagi at broadcom.com>; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel
Quoted 11 lines · Anand Nataraja Sundaram
Just wanted to understand more on your concerns on the mods done to Linux NVMe.
The whole work was to tunnel IB protocol over existing NVMe protocol. To do this we first made sure NVMe stack (host, target) is able to send block traffic and non-block (object based ) traffic. To do this, no changes were required in the NVMe protocol itself. Only the target stack needed some modifications to vector
(a) NVMe block traffic to backend NVMe Namespace block driver
(b) non-block IB protocol traffic to RFC transport layer
Quoted 4 lines
The NVMe changes are restricted to below:
drivers/nvme/target/fc.c | 94 +- drivers/nvme/target/io-cmd.c | 44 +- include/linux/nvme-fc-driver.h | 6 +
You forgot the larger chunks of Linux NVMe code you copied while stripping the copyrights and incorrectly relicensing it to a BSD-like license.
The point is that IFF you really want to do RDMA over NVMe you need to defined a new NVMe I/O command set for it and get it standardized. If that is done we could do a proper upper level protocol interface for it, instead of just hacking it into the protocol and code through the backdoor. But as said before there is no upside of using NVMe, I can see the interest in layering on top of FCP to reuse existing hardware accelerations, similar to how NVMe layers on top of FCP for that reason, but there isn't really any value in throwing in another NVMe layer.
PATCH 1/1 RDMA over Fibre Channel
In reply to Muneendra Kumar MQuoted 5 lines · Muneendra Kumar M
Although we concur with the idea of RDMA directly over Fibre channel, the actual implementation addressing the above reasons requires standardization and coordination with FC HBA vendors and other SAN ecosystem players. This effort is ongoing within our organization (Brocade at Broadcom).
What makes you think that a new standardization effort would have to be launched to make RDMA over FC possible? RDMA over FC was standardized a long time ago and is called "FC-VI". Why aren't you using that standard? A quote from the FC-VI document:
Table 1 ? VI and FC-VI functions
................................
VI function FC-VI equivalent VI Message Exchange Request/Response primitives Sequence VI Connect/Disconnect request Unsolicited control IU VI Connect response Solicited control IU VI Send Data Solicited data IU VI Send Response Command status IU VI RDMA Write Request Solicited data IU VI RDMA Write Response Command status IU VI RDMA Read Request Unsolicited command IU VI RDMA Read Response Solicited data IU
See also ANSI/INCITS 357:2002, "Fibre Channel Virtual Interface Architecture Mapping Protocol (FC-VI)", NCITS Project 1332-D, July 2000.
Bart.