nvme-fabrics: Add target support for FC transport
Subsystems (3)
PATCH v3 6/7 nvme-fabrics: Add target support for FC transport
Add nvme-fabrics target support for FC transport
Implements the FC-NVME T11 definition of how nvme fabric capsules are performed on an FC fabric. Utilizes a lower-layer API to FC host adapters to send/receive FC-4 LS operations and perform the FCP transactions necessary to perform and FCP IO request for NVME.
The T11 definitions for FC-4 Link Services are implemented which create NVMeOF connections. Implements the hooks with nvmet layer to pass NVME commands to it for processing and posting of data/response base to the host via the different connections.
--- MAINTAINERS | 1 + drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/fc.c | 2050 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 2063 insertions(+) create mode 100644 drivers/nvme/target/fc.c
MAINTAINERS +1 -0
@@ -8666,6 +8666,7 @@ S: Supported F: include/linux/nvme-fc.h F: include/linux/nvme-fc-driver.h F: drivers/nvme/host/fc.c +F: drivers/nvme/target/fc.c NVMEM FRAMEWORK M: Srinivas Kandagatla <srinivas.kandagatla at linaro.org>
drivers/nvme/target/Kconfig +10 -0
@@ -34,3 +34,13 @@ config NVME_TARGET_RDMA devices over RDMA. If unsure, say N. + +config NVME_TARGET_FC + tristate "NVMe over Fabrics FC target driver" + depends on NVME_TARGET + help + This enables the NVMe FC target support, which allows exporting NVMe + devices over FC. + + If unsure, say N. +
drivers/nvme/target/Makefile +2 -0
@@ -2,8 +2,10 @@ obj-$(CONFIG_NVME_TARGET) += nvmet.o obj-$(CONFIG_NVME_TARGET_LOOP) += nvme-loop.o obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o +obj-$(CONFIG_NVME_TARGET_FC) += nvmet-fc.o nvmet-y += core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o \ discovery.o nvme-loop-y += loop.o nvmet-rdma-y += rdma.o +nvmet-fc-y += fc.o
drivers/nvme/target/fc.c new file +2050 -0
2,050 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH v3 6/7 nvme-fabrics: Add target support for FC transport
Quoted 105 lines
drivers/nvme/target/fc.c new file +13
-0+static void +nvmet_fc_xmt_fcp_op_done (struct nvmefc_tgt_fcp_req *fcpreq) +{ +struct nvmet_fc_fcp_iod *fod = fcpreq->nvmet_fc_private; +struct nvmet_fc_tgtport *tgtport = fod->tgtport; +struct nvme_fc_ersp_iu *ersp = &fod->rspiubuf; +struct nvme_completion *cqe = &ersp->cqe; + +switch (fcpreq->op) { + +case NVMET_FCOP_WRITEDATA : +if (fcpreq->fcp_error) { + nvmet_req_complete(&fod->req, fcpreq-Quoted 1 line
fcp_error);drivers/nvme/target/fc.c new file +3
-0+return ; + } + if (fcpreq->transferred_length != fcpreq-Quoted 1 line
transfer_length) {drivers/nvme/target/fc.c new file +10
-0+nvmet_req_complete (&fod->req, +NVME_SC_FC_TRANSPORT_ERROR ); +return ; + } + + fod->offset += fcpreq->transferred_length; +if (fod->offset != fod->total_length) { +/* transfer the next chunk */ +nvmet_fc_transfer_fcp_data (tgtport, fod, + NVMET_FCOP_WRITEDATA);
drivers/nvme/target/fc.c new file +4
-0+return ; + } + +/* data transfer complete, resume with nvmet layer */
drivers/nvme/target/fc.c new file +9
-0+ + fod->req.execute (&fod->req); + +break ; + +case NVMET_FCOP_READDATA : +if (fcpreq->fcp_error) { +/* overwrite the nvmet status */ + cqe->status = cpu_to_le16(fcpreq-Quoted 1 line
fcp_error);drivers/nvme/target/fc.c new file +4
-0+ }else if (fcpreq->transferred_length != + fcpreq->transfer_length) { +/* overwrite the nvmet status */ + cqe->status =
cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR);drivers/nvme/target/fc.c new file +5
-0+ }else { + fod->offset += fcpreq->transferred_length; +if (fod->offset != fod->total_length) { +/* transfer the next chunk */ + nvmet_fc_transfer_fcp_data(tgtport,fod,
(quoted diff) +1
-0+ NVMET_FCOP_READDATA);
drivers/nvme/target/fc.c new file +20
-0+return ; + } + } + +/* data transfer complete, send response */ + +/* data no longer needed */ +nvmet_fc_free_tgt_pgs (fod); + +if (unlikely (atomic_read (&fod->aborted))) +nvmet_fc_abort_op (tgtport, fod->fcpreq); +else +nvmet_fc_xmt_fcp_rsp (tgtport, fod); + +break ; + +case NVMET_FCOP_READDATA_RSP : +if (fcpreq->fcp_error) { +/* overwrite the nvmet status */ + cqe->status = cpu_to_le16(fcpreq-Quoted 1 line
fcp_error);drivers/nvme/target/fc.c new file +4
-0+ }else if (fcpreq->transferred_length != + fcpreq->transfer_length) { +/* overwrite the nvmet status */ + cqe->status =
cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR);drivers/nvme/target/fc.c new file +4
-0+ }else + fod->offset += fcpreq->transferred_length; + +/* data transfer complete, response complete as well */
drivers/nvme/target/fc.c new file +6
-0+ +/* data no longer needed */ +nvmet_fc_free_tgt_pgs (fod); + + dma_sync_single_for_cpu(tgtport->dev, fod->rspdma, + sizeof(fod->rspiubuf),
DMA_TO_DEVICE);drivers/nvme/target/fc.c new file +8
-0+nvmet_fc_free_fcp_iod (tgtport, fod); +nvmet_fc_tgt_q_put (fod->queue); +break ; + +case NVMET_FCOP_RSP : +case NVMET_FCOP_ABORT : + dma_sync_single_for_cpu(tgtport->dev, fod->rspdma, + sizeof(fod->rspiubuf),
DMA_TO_DEVICE);
drivers/nvme/target/fc.c line 1679
Quoted 7 lines
drivers/nvme/target/fc.c new file +7
-0+nvmet_fc_free_fcp_iod (tgtport, fod); +nvmet_fc_tgt_q_put (fod->queue); +break ; + +default : +atomic_set (&fod->aborted,1 ); +nvmet_fc_abort_op (tgtport, fod->fcpreq);
My brain zeroed in on the *abort() functionality, which I don't quite understand. ?Apologies for the basic question here but is this correct that nvmet_fc_abort_op() is *NOT* called for the NVMET_FCOP_ABORT case but nvmet_fc_abort_op() *IS* called for the default case?
Other than that it looks ok,
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v3 6/7 nvme-fabrics: Add target support for FC transport
Quoted 14 lines
nvme_fc_abort_op()would not be called in the normal case.
nvme_fc_abort_op()is called on exception cases. It's called to request the LLDD to kill the exchange (the thing the initiator and target are using for the io) mid processing due to some error.? The cases where it's called are:
When queues are deleted, thus every outstanding io on that queue needs to be implicitly terminated.
If the LLDD refused to send data for the exchange. Something is bad in the LLDD or the exchange is dead. So we formally terminate the exchange as there's nothing more the transport can do with it. On the other end of the wire, the initiator won't see the io complete, thus it'll usually fall into the escalation policy of terminating the
Quoted 9 lines
queue or resetting the controller.
if the transport detects a software error for the io that had a bad op code in it. At least kill the FC resource so the initiator will see it and recover.
The call tonvmet_fc_xmt_fcp_op_done()with the ABORT op, is the LLDD saying it's completed the abort, freed the hw resource, and the job/exchange is released back to the transport. At that point, the transport will finish its teardown of the io.
Thanks for the explanation James!
Jay
Quoted 5 lines
Thanks
-- james