nvme-fc/nvmet-fc: Add FC-NVME-2 disconnect association support
Subsystems (5)
cover nvme-fc/nvmet-fc: Add FC-NVME-2 disconnect association support
At the tail end of FC-NVME (1) standardization, the process for terminating an association was changed requiring interlock using FC-NVME Disconnect Assocation LS's between both the host port and target port. This was immediately relaxed with an ammendment to FC-NVME (1) and with wording put into FC-NVME-2. The interlock was removed, but it still required both the host port and target port to initiate Disconnect Association LS's and respond to LS's.
The linux nvme-fc and nvmet-fc implementations were interoperable with standards but the linux-driver api did not support all the functionality needed. It was missing:
- nvme-fc: didn't support the reception of NVME LS's and the ability to transmit responses to an LS.
- nvmet-fc: didn't support the ability to send an NVME LS request. It also did not support a method for the transport to specify a remote port for an LS.
This patch adds the missing functionality. Specifically the patch set:
- Updates the header with the FC-NVME-2 standard out for final approval.
- Refactors data structure names that used to be dependent on role (ls requests were specific to nvme; ls responses were specific to nvmet) to generic names that can be used by both nvme-fc and nvmet-fc.
- Modifies the nvme-fc transport template with interfaces to receive NVME LS's and for the transport to then request LS responses to be sent.
- Modifies the nvmet-fc transport template with:
- The current NVME LS receive interface was modified to supply a handle to indentify the remote port the LS as received from. If the LS creates an association, the handle may be used to initiate NVME LS requests to the remote port. An interface was put in place to invalidate the handle on connectivity losses.
- Interfaces for the transport to request an NVME LS request to be performed as well as to abort that LS in cases of error/teardown.
- The nvme-fc transport was modified to follow the standard:
- Disconnect association logic was revised to send Disconnect LS as soon as all ABTS's were transmit rather than waiting for the ABTS process to fully complete.
- Disconnect LS reception is supported, with reception initiating controller reset and reconnect.
- Disconnect LS responses will not be transmit until association termination has transmit the Disconnect LS.
- The nvmet-fc transport was modified to follow the standard:
- Disconnect assocation logic was revised to transmit a Disconnect LS request as soon as all ABTS's have been transmit. In the past, no Disconnect LS had been transmit.
- Disconnect LS responses will not be sent until the Disconnect LS request has been transmit.
- nvme-fcloop: was updated with interfaces to allow testing of the transports.
- Along the way, cleanups and slight corrections were made to the transports.
- The lpfc driver was modified to support the new transport interfaces for both the nvme and nvmet transports. As much of the functionality was already present, but specific to one side of the transport, existing code was refactored to create common routines. Addition of the new interfaces was able to slip in rather easily with the common routines.
This code was cut against the for-5.6 branch.
I'll work with Martin to minimize any work to merge these lpfc mods with lpfc changes in the scsi tree.
-- james
James Smart (29):
nvme-fc: Sync header to FC-NVME-2 rev 1.08
nvmet-fc: fix typo in comment
nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request nvme-fc nvmet_fc nvme_fcloop: adapt code to changed names in api
header
lpfc: adapt code to changed names in api header
nvme-fcloop: Fix deallocation of working context
nvme-fc nvmet-fc: refactor for common LS definitions
nvmet-fc: Better size LS buffers
nvme-fc: Ensure private pointers are NULL if no data
nvmefc: Use common definitions for LS names, formatting, and
validation
nvme-fc: convert assoc_active flag to atomic
nvme-fc: Add Disconnect Association Rcv support
nvmet-fc: add LS failure messages
nvmet-fc: perform small cleanups on unneeded checks
nvmet-fc: track hostport handle for associations
nvmet-fc: rename ls_list to ls_rcv_list
nvmet-fc: Add Disconnect Association Xmt support
nvme-fcloop: refactor to enable target to host LS
nvme-fcloop: add target to host LS request support
lpfc: Refactor lpfc nvme headers
lpfc: Refactor nvmet_rcv_ctx to create lpfc_async_xchg_ctx
lpfc: Commonize lpfc_async_xchg_ctx state and flag definitions
lpfc: Refactor NVME LS receive handling
lpfc: Refactor Send LS Request support
lpfc: Refactor Send LS Abort support
lpfc: Refactor Send LS Response support
lpfc: nvme: Add Receive LS Request and Send LS Response support to
nvme
lpfc: nvmet: Add support for NVME LS request hosthandle
lpfc: nvmet: Add Send LS Request and Abort LS Request support
drivers/nvme/host/fc.c | 555 +++++++++++++++++++----- drivers/nvme/host/fc.h | 227 ++++++++++ drivers/nvme/target/fc.c | 800 +++++++++++++++++++++++++---------- drivers/nvme/target/fcloop.c | 228 ++++++++-- drivers/scsi/lpfc/lpfc.h | 2 +- drivers/scsi/lpfc/lpfc_attr.c | 3 - drivers/scsi/lpfc/lpfc_crtn.h | 9 +- drivers/scsi/lpfc/lpfc_ct.c | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 5 +- drivers/scsi/lpfc/lpfc_hbadisc.c | 8 +- drivers/scsi/lpfc/lpfc_init.c | 7 +- drivers/scsi/lpfc/lpfc_mem.c | 4 - drivers/scsi/lpfc/lpfc_nportdisc.c | 13 +- drivers/scsi/lpfc/lpfc_nvme.c | 550 ++++++++++++++++-------- drivers/scsi/lpfc/lpfc_nvme.h | 198 +++++++++ drivers/scsi/lpfc/lpfc_nvmet.c | 837 +++++++++++++++++++++++-------------- drivers/scsi/lpfc/lpfc_nvmet.h | 158 ------- drivers/scsi/lpfc/lpfc_sli.c | 126 +++++- include/linux/nvme-fc-driver.h | 368 +++++++++++----- include/linux/nvme-fc.h | 9 +- 20 files changed, 2970 insertions(+), 1138 deletions(-) create mode 100644 drivers/nvme/host/fc.h delete mode 100644 drivers/scsi/lpfc/lpfc_nvmet.h
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 1/29 nvme-fc: Sync header to FC-NVME-2 rev 1.08
A couple of minor changes occurred between 1.06 and 1.08:
- Addition of NVME_SR_RSP opcode
- change of SR_RSP status code 1 to Reserved
--- include/linux/nvme-fc.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
include/linux/nvme-fc.h +5 -4
@@ -4,8 +4,8 @@*/ /* -* This file contains definitions relative to FC-NVME-2 r1.06 -* (T11-2019-00210-v001). +* This file contains definitions relative to FC-NVME-2 r1.08 +* (T11-2019-00210-v004). */ #ifndef _NVME_FC_H
@@ -81,7 +81,8 @@ struct nvme_fc_ersp_iu { }; -#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_RSP_OPCODE 0x02struct nvme_fc_nvme_sr_iu {__u8 fc_id;
@@ -94,7 +95,7 @@ struct nvme_fc_nvme_sr_iu {enum {FCNVME_SRSTAT_ACC =0x0 , -FCNVME_SRSTAT_INV_FCID =0x1 , +/* reserved 0x1 */ /* reserved 0x2 */ FCNVME_SRSTAT_LOGICAL_ERR =0x3 ,FCNVME_SRSTAT_INV_QUALIF =0x4 ,
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 01/29] nvme-fc: Sync header to FC-NVME-2 rev 1.08
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 01/29] nvme-fc: Sync header to FC-NVME-2 rev 1.08
In reply to James SmartQuoted 36 lines · James Smart
A couple of minor changes occurred between 1.06 and 1.08:
- Addition ofNVME_SR_RSPopcode
- change ofSR_RSPstatus code 1 to ReservedSigned-off-by James Smart <[email protected]>--- include/linux/nvme-fc.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)include/linux/nvme-fc.h +4
-3@@ -4,8 +4,8 @@*/ /* -* This file contains definitions relative to FC-NVME-2 r1.06 -* (T11-2019-00210-v001). +* This file contains definitions relative to FC-NVME-2 r1.08 +* (T11-2019-00210-v004). */ #ifndef _NVME_FC_H@@ -81,7 +81,8 @@ struct nvme_fc_ersp_iu { }; -#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_RSP_OPCODE 0x02struct nvme_fc_nvme_sr_iu {__u8 fc_id;
include/linux/nvme-fc.h struct nvme_fc_nvme_sr_iu { · line 101
Quoted 8 lines
include/linux/nvme-fc.h +1
-1@@ -94,7 +95,7 @@ struct nvme_fc_nvme_sr_iu {enum {FCNVME_SRSTAT_ACC =0x0 , -FCNVME_SRSTAT_INV_FCID =0x1 , +/* reserved 0x1 */ /* reserved 0x2 */ FCNVME_SRSTAT_LOGICAL_ERR =0x3 ,FCNVME_SRSTAT_INV_QUALIF =0x4 ,
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 01/29] nvme-fc: Sync header to FC-NVME-2 rev 1.08
In reply to James SmartQuoted 36 lines · James Smart
A couple of minor changes occurred between 1.06 and 1.08:
- Addition ofNVME_SR_RSPopcode
- change ofSR_RSPstatus code 1 to ReservedSigned-off-by James Smart <[email protected]>--- include/linux/nvme-fc.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)include/linux/nvme-fc.h +4
-3@@ -4,8 +4,8 @@*/ /* -* This file contains definitions relative to FC-NVME-2 r1.06 -* (T11-2019-00210-v001). +* This file contains definitions relative to FC-NVME-2 r1.08 +* (T11-2019-00210-v004). */ #ifndef _NVME_FC_H@@ -81,7 +81,8 @@ struct nvme_fc_ersp_iu { }; -#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_OPCODE 0x01 +#define FCNVME_NVME_SR_RSP_OPCODE 0x02struct nvme_fc_nvme_sr_iu {__u8 fc_id;
include/linux/nvme-fc.h struct nvme_fc_nvme_sr_iu { · line 101
Quoted 8 lines
include/linux/nvme-fc.h +1
-1@@ -94,7 +95,7 @@ struct nvme_fc_nvme_sr_iu {enum {FCNVME_SRSTAT_ACC =0x0 , -FCNVME_SRSTAT_INV_FCID =0x1 , +/* reserved 0x1 */ /* reserved 0x2 */ FCNVME_SRSTAT_LOGICAL_ERR =0x3 ,FCNVME_SRSTAT_INV_QUALIF =0x4 ,
Looks Good.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/29 nvmet-fc: fix typo in comment
In reply to James SmartFix typo in comment: about should be abort
--- drivers/nvme/target/fc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/fc.c +1 -1
@@ -684,7 +684,7 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) disconnect =atomic_xchg (&queue->connected,0 );spin_lock_irqsave (&queue->qlock, flags); -/* about outstanding io's */ +/* abort outstanding io's */ for (i =0 ; i < queue->sqsize; fod++, i++) {if (fod->active) {spin_lock (&fod->flock);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 02/29] nvmet-fc: fix typo in comment
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 02/29] nvmet-fc: fix typo in comment
In reply to James SmartQuoted 13 lines · James Smart
Fix typo in comment: about should be abort
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/target/fc.c +0
-0
drivers/nvme/target/fc.c nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) · line 690
Quoted 8 lines
drivers/nvme/target/fc.c +1
-1@@ -684,7 +684,7 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) disconnect =atomic_xchg (&queue->connected,0 );spin_lock_irqsave (&queue->qlock, flags); -/* about outstanding io's */ +/* abort outstanding io's */ for (i =0 ; i < queue->sqsize; fod++, i++) {if (fod->active) {spin_lock (&fod->flock);
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 02/29] nvmet-fc: fix typo in comment
In reply to James SmartQuoted 13 lines · James Smart
Fix typo in comment: about should be abort
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/target/fc.c +0
-0
drivers/nvme/target/fc.c nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) · line 690
Quoted 8 lines
drivers/nvme/target/fc.c +1
-1@@ -684,7 +684,7 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) disconnect =atomic_xchg (&queue->connected,0 );spin_lock_irqsave (&queue->qlock, flags); -/* about outstanding io's */ +/* abort outstanding io's */ for (i =0 ; i < queue->sqsize; fod++, i++) {if (fod->active) {spin_lock (&fod->flock);
Looks Good.
Signature · 6 lines
- Himanshu _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 3/29 nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request
In reply to James SmartThe current LLDD api has:
nvme-fc: contains api for transport to do LS requests (and aborts of
them). However, there is no interface for reception of LS's and sending responses for them.
nvmet-fc: contains api for transport to do reception of LS's and sending
of responses for them. However, there is no interface for doing LS requests.
Revise the api's so that both nvme-fc and nvmet-fc can send LS's, as well as receiving LS's and sending their responses.
Change name of the rcv_ls_req struct to better reflect generic use as a context to used to send an ls rsp.
Change nvmet_fc_rcv_ls_req() calling sequence to provide handle that can be used by transport in later LS request sequences for an association.
--- include/linux/nvme-fc-driver.h | 368 ++++++++++++++++++++++++++++++----------- 1 file changed, 270 insertions(+), 98 deletions(-)
include/linux/nvme-fc-driver.h +270 -98
@@ -10,47 +10,26 @@/* -* ********************** LLDD FC-NVME Host API ******************** +* ********************** FC-NVME LS API ******************** * -* For FC LLDD's that are the NVME Host role. +* Data structures used by both FC-NVME hosts and FC-NVME +* targets to perform FC-NVME LS requests or transmit +* responses. * -* ****************************************************************** +* *********************************************************** */ - -/** -* struct nvme_fc_port_info - port-specific ids and FC connection-specific -* data element used during NVME Host role -* registrations -* -* Static fields describing the port being registered: -* @node_name: FC WWNN for the port -* @port_name: FC WWPN for the port -* @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx) -* @dev_loss_tmo: maximum delay for reconnects to an association on -* this device. Used only on a remoteport. +* struct nvmefc_ls_req - Request structure passed from the transport +* to the LLDD to perform a NVME-FC LS request and obtain +* a response. +* Used by nvme-fc transport (host) to send LS's such as +* Create Association, Create Connection and Disconnect +* Association. +* Used by the nvmet-fc transport (controller) to send +* LS's such as Disconnect Association. * -* Initialization values for dynamic port fields: -* @port_id: FC N_Port_ID currently assigned the port. Upper 8 bits must -* be set to 0. -*/ -struct nvme_fc_port_info { -u64 node_name; -u64 port_name; -u32 port_role; -u32 port_id; -u32 dev_loss_tmo; -}; - - -/** -* struct nvmefc_ls_req - Request structure passed from NVME-FC transport -* to LLDD in order to perform a NVME FC-4 LS -* request and obtain a response. -* -* Values set by the NVME-FC layer prior to calling the LLDD ls_req -* entrypoint. +* Values set by the requestor prior to calling the LLDD ls_req entrypoint: * @rqstaddr: pointer to request buffer * @rqstdma: PCI DMA address of request buffer * @rqstlen: Length, in bytes, of request buffer
@@ -63,8 +42,8 @@ struct nvme_fc_port_info {* @private: pointer to memory allocated alongside the ls request structure * that is specifically for the LLDD to use while processing the * request. The length of the buffer corresponds to the -* lsrqst_priv_sz value specified in the nvme_fc_port_template -* supplied by the LLDD. +* lsrqst_priv_sz value specified in the xxx_template supplied +* by the LLDD. * @done: The callback routine the LLDD is to invoke upon completion of * the LS request. req argument is the pointer to the original LS * request structure. Status argument must be 0 upon success, a
@@ -86,6 +65,101 @@ struct nvmefc_ls_req { }__aligned(sizeof(u64)) ;/* alignment for other things alloc'd with */ +/** +* struct nvmefc_ls_rsp - Structure passed from the transport to the LLDD +* to request the transmit the NVME-FC LS response to a +* NVME-FC LS request. The structure originates in the LLDD +* and is given to the transport via the xxx_rcv_ls_req() +* transport routine. As such, the structure represents the +* FC exchange context for the NVME-FC LS request that was +* received and which the response is to be sent for. +* Used by the LLDD to pass the nvmet-fc transport (controller) +* received LS's such as Create Association, Create Connection +* and Disconnect Association. +* Used by the LLDD to pass the nvme-fc transport (host) +* received LS's such as Disconnect Association or Disconnect +* Connection. +* +* The structure is allocated by the LLDD whenever a LS Request is received +* from the FC link. The address of the structure is passed to the nvmet-fc +* or nvme-fc layer via the xxx_rcv_ls_req() transport routines. +* +* The address of the structure is to be passed back to the LLDD +* when the response is to be transmit. The LLDD will use the address to +* map back to the LLDD exchange structure which maintains information such +* the remote N_Port that sent the LS as well as any FC exchange context. +* Upon completion of the LS response transmit, the LLDD will pass the +* address of the structure back to the transport LS rsp done() routine, +* allowing the transport release dma resources. Upon completion of +* the done() routine, no further access to the structure will be made by +* the transport and the LLDD can de-allocate the structure. +* +* Field initialization: +* At the time of the xxx_rcv_ls_req() call, there is no content that +* is valid in the structure. +* +* When the structure is used for the LLDD->xmt_ls_rsp() call, the +* transport layer will fully set the fields in order to specify the +* response payload buffer and its length as well as the done routine +* to be called upon completion of the transmit. The transport layer +* will also set a private pointer for its own use in the done routine. +* +* Values set by the transport layer prior to calling the LLDD xmt_ls_rsp +* entrypoint: +* @rspbuf: pointer to the LS response buffer +* @rspdma: PCI DMA address of the LS response buffer +* @rsplen: Length, in bytes, of the LS response buffer +* @done: The callback routine the LLDD is to invoke upon completion of +* transmitting the LS response. req argument is the pointer to +* the original ls request. +* @nvme_fc_private: pointer to an internal transport-specific structure +* used as part of the transport done() processing. The LLDD is +* not to access this pointer. +*/ +struct nvmefc_ls_rsp { +void *rspbuf; +dma_addr_t rspdma; +u16 rsplen; + +void (*done)(struct nvmefc_ls_rsp *rsp); +void *nvme_fc_private;/* LLDD is not to access !! */ +}; + + + +/* +* ********************** LLDD FC-NVME Host API ******************** +* +* For FC LLDD's that are the NVME Host role. +* +* ****************************************************************** +*/ + + +/** +* struct nvme_fc_port_info - port-specific ids and FC connection-specific +* data element used during NVME Host role +* registrations +* +* Static fields describing the port being registered: +* @node_name: FC WWNN for the port +* @port_name: FC WWPN for the port +* @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx) +* @dev_loss_tmo: maximum delay for reconnects to an association on +* this device. Used only on a remoteport. +* +* Initialization values for dynamic port fields: +* @port_id: FC N_Port_ID currently assigned the port. Upper 8 bits must +* be set to 0. +*/ +struct nvme_fc_port_info { +u64 node_name; +u64 port_name; +u32 port_role; +u32 port_id; +u32 dev_loss_tmo; +}; +enum nvmefc_fcp_datadir {NVMEFC_FCP_NODATA ,/* payload_length and sg_cnt will be zero */ NVMEFC_FCP_WRITE ,
@@ -339,6 +413,21 @@ struct nvme_fc_remote_port {* indicating an FC transport Aborted status. * Entrypoint is Mandatory. * +* @xmt_ls_rsp: Called to transmit the response to a FC-NVME FC-4 LS service. +* The nvmefc_ls_rsp structure is the same LLDD-supplied exchange +* structure specified in the nvme_fc_rcv_ls_req() call made when +* the LS request was received. The structure will fully describe +* the buffers for the response payload and the dma address of the +* payload. The LLDD is to transmit the response (or return a +* non-zero errno status), and upon completion of the transmit, call +* the "done" routine specified in the nvmefc_ls_rsp structure +* (argument to done is the address of the nvmefc_ls_rsp structure +* itself). Upon the completion of the done routine, the LLDD shall +* consider the LS handling complete and the nvmefc_ls_rsp structure +* may be freed/released. +* Entrypoint is mandatory if the LLDD calls the nvme_fc_rcv_ls_req() +* entrypoint. +* * @max_hw_queues: indicates the maximum number of hw queues the LLDD * supports for cpu affinitization. * Value is Mandatory. Must be at least 1.
@@ -373,7 +462,7 @@ struct nvme_fc_remote_port {* @lsrqst_priv_sz: The LLDD sets this field to the amount of additional * memory that it would like fc nvme layer to allocate on the LLDD's * behalf whenever a ls request structure is allocated. The additional -* memory area solely for the of the LLDD and its location is +* memory area is solely for use by the LLDD and its location is * specified by the ls_request->private pointer. * Value is Mandatory. Allowed to be zero. *
@@ -409,6 +498,9 @@ struct nvme_fc_port_template { struct nvme_fc_remote_port *,void *hw_queue_handle, struct nvmefc_fcp_req *); +int (*xmt_ls_rsp)(struct nvme_fc_local_port *localport, +struct nvme_fc_remote_port *rport, +struct nvmefc_ls_rsp *ls_rsp);u32 max_hw_queues;u16 max_sgl_segments;
@@ -445,6 +537,34 @@ void nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport);int nvme_fc_set_remoteport_devloss (struct nvme_fc_remote_port *remoteport,u32 dev_loss_tmo); +/* +* Routine called to pass a NVME-FC LS request, received by the lldd, +* to the nvme-fc transport. +* +* If the return value is zero: the LS was successfully accepted by the +* transport. +* If the return value is non-zero: the transport has not accepted the +* LS. The lldd should ABTS-LS the LS. +* +* Note: if the LLDD receives and ABTS for the LS prior to the transport +* calling the ops->xmt_ls_rsp() routine to transmit a response, the LLDD +* shall mark the LS as aborted, and when the xmt_ls_rsp() is called: the +* response shall not be transmit and the struct nvmefc_ls_rsp() done +* routine shall be called. The LLDD may transmit the ABTS response as +* soon as the LS was marked or can delay until the xmt_ls_rsp() call is +* made. +* Note: if an RCV LS was successfully posted to the transport and the +* remoteport is then unregistered before xmt_ls_rsp() was called for +* the lsrsp structure, the transport will still call xmt_ls_rsp() +* afterward to cleanup the outstanding lsrsp structure. The LLDD should +* noop the transmission of the rsp and call the lsrsp->done() routine +* to allow the lsrsp structure to be released. +*/ +int nvme_fc_rcv_ls_req (struct nvme_fc_remote_port *remoteport, +struct nvmefc_ls_rsp *lsrsp, +void *lsreqbuf,u32 lsreqbuf_len); + +/* * *************** LLDD FC-NVME Target/Subsystem API ***************
@@ -474,55 +594,6 @@ struct nvmet_fc_port_info { }; -/** -* struct nvmefc_tgt_ls_req - Structure used between LLDD and NVMET-FC -* layer to represent the exchange context for -* a FC-NVME Link Service (LS). -* -* The structure is allocated by the LLDD whenever a LS Request is received -* from the FC link. The address of the structure is passed to the nvmet-fc -* layer via the nvmet_fc_rcv_ls_req() call. The address of the structure -* will be passed back to the LLDD when the response is to be transmit. -* The LLDD is to use the address to map back to the LLDD exchange structure -* which maintains information such as the targetport the LS was received -* on, the remote FC NVME initiator that sent the LS, and any FC exchange -* context. Upon completion of the LS response transmit, the address of the -* structure will be passed back to the LS rsp done() routine, allowing the -* nvmet-fc layer to release dma resources. Upon completion of the done() -* routine, no further access will be made by the nvmet-fc layer and the -* LLDD can de-allocate the structure. -* -* Field initialization: -* At the time of the nvmet_fc_rcv_ls_req() call, there is no content that -* is valid in the structure. -* -* When the structure is used for the LLDD->xmt_ls_rsp() call, the nvmet-fc -* layer will fully set the fields in order to specify the response -* payload buffer and its length as well as the done routine to be called -* upon compeletion of the transmit. The nvmet-fc layer will also set a -* private pointer for its own use in the done routine. -* -* Values set by the NVMET-FC layer prior to calling the LLDD xmt_ls_rsp -* entrypoint. -* @rspbuf: pointer to the LS response buffer -* @rspdma: PCI DMA address of the LS response buffer -* @rsplen: Length, in bytes, of the LS response buffer -* @done: The callback routine the LLDD is to invoke upon completion of -* transmitting the LS response. req argument is the pointer to -* the original ls request. -* @nvmet_fc_private: pointer to an internal NVMET-FC layer structure used -* as part of the NVMET-FC processing. The LLDD is not to access -* this pointer. -*/ -struct nvmefc_tgt_ls_req { -void *rspbuf; -dma_addr_t rspdma; -u16 rsplen; - -void (*done)(struct nvmefc_tgt_ls_req *req); -void *nvmet_fc_private;/* LLDD is not to access !! */ -}; -/* Operations that NVME-FC layer may request the LLDD to perform for FCP */ enum {NVMET_FCOP_READDATA =1 ,/* xmt data to initiator */
@@ -697,17 +768,19 @@ struct nvmet_fc_target_port {* Entrypoint is Mandatory. * * @xmt_ls_rsp: Called to transmit the response to a FC-NVME FC-4 LS service. -* The nvmefc_tgt_ls_req structure is the same LLDD-supplied exchange +* The nvmefc_ls_rsp structure is the same LLDD-supplied exchange * structure specified in the nvmet_fc_rcv_ls_req() call made when -* the LS request was received. The structure will fully describe +* the LS request was received. The structure will fully describe * the buffers for the response payload and the dma address of the -* payload. The LLDD is to transmit the response (or return a non-zero -* errno status), and upon completion of the transmit, call the -* "done" routine specified in the nvmefc_tgt_ls_req structure -* (argument to done is the ls reqwuest structure itself). -* After calling the done routine, the LLDD shall consider the -* LS handling complete and the nvmefc_tgt_ls_req structure may -* be freed/released. +* payload. The LLDD is to transmit the response (or return a +* non-zero errno status), and upon completion of the transmit, call +* the "done" routine specified in the nvmefc_ls_rsp structure +* (argument to done is the address of the nvmefc_ls_rsp structure +* itself). Upon the completion of the done() routine, the LLDD shall +* consider the LS handling complete and the nvmefc_ls_rsp structure +* may be freed/released. +* The transport will always call the xmt_ls_rsp() routine for any +* LS received. * Entrypoint is Mandatory. * * @fcp_op: Called to perform a data transfer or transmit a response.
@@ -802,6 +875,39 @@ struct nvmet_fc_target_port {* should cause the initiator to rescan the discovery controller * on the targetport. * +* @ls_req: Called to issue a FC-NVME FC-4 LS service request. +* The nvme_fc_ls_req structure will fully describe the buffers for +* the request payload and where to place the response payload. +* The targetport that is to issue the LS request is identified by +* the targetport argument. The remote port that is to receive the +* LS request is identified by the hosthandle argument. The nvmet-fc +* transport is only allowed to issue FC-NVME LS's on behalf of an +* association that was created prior by a Create Association LS. +* The hosthandle will originate from the LLDD in the struct +* nvmefc_ls_rsp structure for the Create Association LS that +* was delivered to the transport. The transport will save the +* hosthandle as an attribute of the association. If the LLDD +* loses connectivity with the remote port, it must call the +* nvmet_fc_invalidate_host() routine to remove any references to +* the remote port in the transport. +* The LLDD is to allocate an exchange, issue the LS request, obtain +* the LS response, and call the "done" routine specified in the +* request structure (argument to done is the ls request structure +* itself). +* Entrypoint is Optional - but highly recommended. +* +* @ls_abort: called to request the LLDD to abort the indicated ls request. +* The call may return before the abort has completed. After aborting +* the request, the LLDD must still call the ls request done routine +* indicating an FC transport Aborted status. +* Entrypoint is Mandatory if the ls_req entry point is specified. +* +* @host_release: called to inform the LLDD that the request to invalidate +* the host port indicated by the hosthandle has been fully completed. +* No associations exist with the host port and there will be no +* further references to hosthandle. +* Entrypoint is Mandatory if the lldd calls nvmet_fc_invalidate_host(). +* * @max_hw_queues: indicates the maximum number of hw queues the LLDD * supports for cpu affinitization. * Value is Mandatory. Must be at least 1.
@@ -830,11 +936,19 @@ struct nvmet_fc_target_port {* area solely for the of the LLDD and its location is specified by * the targetport->private pointer. * Value is Mandatory. Allowed to be zero. +* +* @lsrqst_priv_sz: The LLDD sets this field to the amount of additional +* memory that it would like nvmet-fc layer to allocate on the LLDD's +* behalf whenever a ls request structure is allocated. The additional +* memory area is solely for use by the LLDD and its location is +* specified by the ls_request->private pointer. +* Value is Mandatory. Allowed to be zero. +* */ struct nvmet_fc_target_template {void (*targetport_delete)(struct nvmet_fc_target_port *tgtport);int (*xmt_ls_rsp)(struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *tls_req); +struct nvmefc_ls_rsp *ls_rsp);int (*fcp_op)(struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq); void (*fcp_abort)(struct nvmet_fc_target_port *tgtport,
@@ -844,6 +958,11 @@ struct nvmet_fc_target_template {void (*defer_rcv)(struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq);void (*discovery_event)(struct nvmet_fc_target_port *tgtport); +int (*ls_req)(struct nvmet_fc_target_port *targetport, +void *hosthandle,struct nvmefc_ls_req *lsreq); +void (*ls_abort)(struct nvmet_fc_target_port *targetport, +void *hosthandle,struct nvmefc_ls_req *lsreq); +void (*host_release)(void *hosthandle);u32 max_hw_queues;u16 max_sgl_segments;
@@ -852,7 +971,9 @@ struct nvmet_fc_target_template {u32 target_features; +/* sizes of additional private data for data structures */ u32 target_priv_sz; +u32 lsrqst_priv_sz; };
@@ -863,10 +984,61 @@ int nvmet_fc_register_targetport(struct nvmet_fc_port_info *portinfo,int nvmet_fc_unregister_targetport (struct nvmet_fc_target_port *tgtport); +/* +* Routine called to pass a NVME-FC LS request, received by the lldd, +* to the nvmet-fc transport. +* +* If the return value is zero: the LS was successfully accepted by the +* transport. +* If the return value is non-zero: the transport has not accepted the +* LS. The lldd should ABTS-LS the LS. +* +* Note: if the LLDD receives and ABTS for the LS prior to the transport +* calling the ops->xmt_ls_rsp() routine to transmit a response, the LLDD +* shall mark the LS as aborted, and when the xmt_ls_rsp() is called: the +* response shall not be transmit and the struct nvmefc_ls_rsp() done +* routine shall be called. The LLDD may transmit the ABTS response as +* soon as the LS was marked or can delay until the xmt_ls_rsp() call is +* made. +* Note: if an RCV LS was successfully posted to the transport and the +* targetport is then unregistered before xmt_ls_rsp() was called for +* the lsrsp structure, the transport will still call xmt_ls_rsp() +* afterward to cleanup the outstanding lsrsp structure. The LLDD should +* noop the transmission of the rsp and call the lsrsp->done() routine +* to allow the lsrsp structure to be released. +*/ int nvmet_fc_rcv_ls_req (struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *lsreq, +void *hosthandle, +struct nvmefc_ls_rsp *rsp,void *lsreqbuf,u32 lsreqbuf_len); +/* +* Routine called by the LLDD whenever it has a logout or loss of +* connectivity to a NVME-FC host port which there had been active +* NVMe controllers for. The host port is indicated by the +* hosthandle. The hosthandle is given to the nvmet-fc transport +* when a NVME LS was received, typically to create a new association. +* The nvmet-fc transport will cache the hostport value with the +* association for use in LS requests for the association. +* When the LLDD calls this routine, the nvmet-fc transport will +* immediately terminate all associations that were created with +* the hosthandle host port. +* The LLDD, after calling this routine and having control returned, +* must assume the transport may subsequently utilize hosthandle as +* part of sending LS's to terminate the association. The LLDD +* should reject the LS's if they are attempted. +* Once the last association has terminated for the hosthandle host +* port, the nvmet-fc transport will call the ops->host_release() +* callback. As of the callback, the nvmet-fc transport will no +* longer reference hosthandle. +*/ +void nvmet_fc_invalidate_host (struct nvmet_fc_target_port *tgtport, +void *hosthandle); + +/* +* If nvmet_fc_rcv_fcp_req returns non-zero, the transport has not accepted +* the FCP cmd. The lldd should ABTS-LS the cmd. +*/ int nvmet_fc_rcv_fcp_req (struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq,void *cmdiubuf,u32 cmdiubuf_len);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 03/29] nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request
Can we have one of the FC guys have a look into this?
I'm not familiar with the details, and it'd be useful to have a second set of eyes on this.
This is general for the whole series.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 03/29] nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request
In reply to James SmartQuoted 14 lines · James Smart
The current LLDD api has:
nvme-fc: contains api for transport to do LS requests (and aborts of
them). However, there is no interface for reception of LS's and sending responses for them.
nvmet-fc: contains api for transport to do reception of LS's and sending
of responses for them. However, there is no interface for doing LS requests.Revise the api's so that both nvme-fc and nvmet-fc can send LS's, as well as receiving LS's and sending their responses.
Change name of the
rcv_ls_reqstruct to better reflect generic use as a context to used to send an ls rsp.
Quoted 8 lines
Change
nvmet_fc_rcv_ls_req()calling sequence to provide handle that can be used by transport in later LS request sequences for an association.Signed-off-by James Smart <[email protected]>--- include/linux/nvme-fc-driver.h | 368 ++++++++++++++++++++++++++++++----------- 1 file changed, 270 insertions(+), 98 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 03/29] nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request
In reply to James SmartQuoted 499 lines · James Smart
The current LLDD api has:
nvme-fc: contains api for transport to do LS requests (and aborts of
them). However, there is no interface for reception of LS's and sending responses for them.
nvmet-fc: contains api for transport to do reception of LS's and sending
of responses for them. However, there is no interface for doing LS requests.Revise the api's so that both nvme-fc and nvmet-fc can send LS's, as well as receiving LS's and sending their responses.
Change name of the
rcv_ls_reqstruct to better reflect generic use as a context to used to send an ls rsp.Change
nvmet_fc_rcv_ls_req()calling sequence to provide handle that can be used by transport in later LS request sequences for an association.Signed-off-by James Smart <[email protected]>--- include/linux/nvme-fc-driver.h | 368 ++++++++++++++++++++++++++++++----------- 1 file changed, 270 insertions(+), 98 deletions(-)include/linux/nvme-fc-driver.h +263
-98@@ -10,47 +10,26 @@/* -* ********************** LLDD FC-NVME Host API ******************** +* ********************** FC-NVME LS API ******************** * -* For FC LLDD's that are the NVME Host role. +* Data structures used by both FC-NVME hosts and FC-NVME +* targets to perform FC-NVME LS requests or transmit +* responses. * -* ****************************************************************** +* *********************************************************** */ - -/** -* struct nvme_fc_port_info - port-specific ids and FC connection-specific -* data element used during NVME Host role -* registrations -* -* Static fields describing the port being registered: -* @node_name: FC WWNN for the port -* @port_name: FC WWPN for the port -* @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx) -* @dev_loss_tmo: maximum delay for reconnects to an association on -* this device. Used only on a remoteport. +* struct nvmefc_ls_req - Request structure passed from the transport +* to the LLDD to perform a NVME-FC LS request and obtain +* a response. +* Used by nvme-fc transport (host) to send LS's such as +* Create Association, Create Connection and Disconnect +* Association. +* Used by the nvmet-fc transport (controller) to send +* LS's such as Disconnect Association. * -* Initialization values for dynamic port fields: -* @port_id: FC N_Port_ID currently assigned the port. Upper 8 bits must -* be set to 0. -*/ -struct nvme_fc_port_info { -u64 node_name; -u64 port_name; -u32 port_role; -u32 port_id; -u32 dev_loss_tmo; -}; - - -/** -* struct nvmefc_ls_req - Request structure passed from NVME-FC transport -* to LLDD in order to perform a NVME FC-4 LS -* request and obtain a response. -* -* Values set by the NVME-FC layer prior to calling the LLDD ls_req -* entrypoint. +* Values set by the requestor prior to calling the LLDD ls_req entrypoint: * @rqstaddr: pointer to request buffer * @rqstdma: PCI DMA address of request buffer * @rqstlen: Length, in bytes, of request buffer @@ -63,8 +42,8 @@ struct nvme_fc_port_info {* @private: pointer to memory allocated alongside the ls request structure * that is specifically for the LLDD to use while processing the * request. The length of the buffer corresponds to the -* lsrqst_priv_sz value specified in the nvme_fc_port_template -* supplied by the LLDD. +* lsrqst_priv_sz value specified in the xxx_template supplied +* by the LLDD. * @done: The callback routine the LLDD is to invoke upon completion of * the LS request. req argument is the pointer to the original LS * request structure. Status argument must be 0 upon success, a @@ -86,6 +65,101 @@ struct nvmefc_ls_req { }__aligned(sizeof(u64)) ;/* alignment for other things alloc'd with */ +/** +* struct nvmefc_ls_rsp - Structure passed from the transport to the LLDD +* to request the transmit the NVME-FC LS response to a +* NVME-FC LS request. The structure originates in the LLDD +* and is given to the transport via the xxx_rcv_ls_req() +* transport routine. As such, the structure represents the +* FC exchange context for the NVME-FC LS request that was +* received and which the response is to be sent for. +* Used by the LLDD to pass the nvmet-fc transport (controller) +* received LS's such as Create Association, Create Connection +* and Disconnect Association. +* Used by the LLDD to pass the nvme-fc transport (host) +* received LS's such as Disconnect Association or Disconnect +* Connection. +* +* The structure is allocated by the LLDD whenever a LS Request is received +* from the FC link. The address of the structure is passed to the nvmet-fc +* or nvme-fc layer via the xxx_rcv_ls_req() transport routines. +* +* The address of the structure is to be passed back to the LLDD +* when the response is to be transmit. The LLDD will use the address to +* map back to the LLDD exchange structure which maintains information such +* the remote N_Port that sent the LS as well as any FC exchange context. +* Upon completion of the LS response transmit, the LLDD will pass the +* address of the structure back to the transport LS rsp done() routine, +* allowing the transport release dma resources. Upon completion of +* the done() routine, no further access to the structure will be made by +* the transport and the LLDD can de-allocate the structure. +* +* Field initialization: +* At the time of the xxx_rcv_ls_req() call, there is no content that +* is valid in the structure. +* +* When the structure is used for the LLDD->xmt_ls_rsp() call, the +* transport layer will fully set the fields in order to specify the +* response payload buffer and its length as well as the done routine +* to be called upon completion of the transmit. The transport layer +* will also set a private pointer for its own use in the done routine. +* +* Values set by the transport layer prior to calling the LLDD xmt_ls_rsp +* entrypoint: +* @rspbuf: pointer to the LS response buffer +* @rspdma: PCI DMA address of the LS response buffer +* @rsplen: Length, in bytes, of the LS response buffer +* @done: The callback routine the LLDD is to invoke upon completion of +* transmitting the LS response. req argument is the pointer to +* the original ls request. +* @nvme_fc_private: pointer to an internal transport-specific structure +* used as part of the transport done() processing. The LLDD is +* not to access this pointer. +*/ +struct nvmefc_ls_rsp { +void *rspbuf; +dma_addr_t rspdma; +u16 rsplen; + +void (*done)(struct nvmefc_ls_rsp *rsp); +void *nvme_fc_private;/* LLDD is not to access !! */ +}; + + + +/* +* ********************** LLDD FC-NVME Host API ******************** +* +* For FC LLDD's that are the NVME Host role. +* +* ****************************************************************** +*/ + + +/** +* struct nvme_fc_port_info - port-specific ids and FC connection-specific +* data element used during NVME Host role +* registrations +* +* Static fields describing the port being registered: +* @node_name: FC WWNN for the port +* @port_name: FC WWPN for the port +* @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx) +* @dev_loss_tmo: maximum delay for reconnects to an association on +* this device. Used only on a remoteport. +* +* Initialization values for dynamic port fields: +* @port_id: FC N_Port_ID currently assigned the port. Upper 8 bits must +* be set to 0. +*/ +struct nvme_fc_port_info { +u64 node_name; +u64 port_name; +u32 port_role; +u32 port_id; +u32 dev_loss_tmo; +}; +enum nvmefc_fcp_datadir {NVMEFC_FCP_NODATA ,/* payload_length and sg_cnt will be zero */ NVMEFC_FCP_WRITE ,@@ -339,6 +413,21 @@ struct nvme_fc_remote_port {* indicating an FC transport Aborted status. * Entrypoint is Mandatory. * +* @xmt_ls_rsp: Called to transmit the response to a FC-NVME FC-4 LS service. +* The nvmefc_ls_rsp structure is the same LLDD-supplied exchange +* structure specified in the nvme_fc_rcv_ls_req() call made when +* the LS request was received. The structure will fully describe +* the buffers for the response payload and the dma address of the +* payload. The LLDD is to transmit the response (or return a +* non-zero errno status), and upon completion of the transmit, call +* the "done" routine specified in the nvmefc_ls_rsp structure +* (argument to done is the address of the nvmefc_ls_rsp structure +* itself). Upon the completion of the done routine, the LLDD shall +* consider the LS handling complete and the nvmefc_ls_rsp structure +* may be freed/released. +* Entrypoint is mandatory if the LLDD calls the nvme_fc_rcv_ls_req() +* entrypoint. +* * @max_hw_queues: indicates the maximum number of hw queues the LLDD * supports for cpu affinitization. * Value is Mandatory. Must be at least 1. @@ -373,7 +462,7 @@ struct nvme_fc_remote_port {* @lsrqst_priv_sz: The LLDD sets this field to the amount of additional * memory that it would like fc nvme layer to allocate on the LLDD's * behalf whenever a ls request structure is allocated. The additional -* memory area solely for the of the LLDD and its location is +* memory area is solely for use by the LLDD and its location is * specified by the ls_request->private pointer. * Value is Mandatory. Allowed to be zero. * @@ -409,6 +498,9 @@ struct nvme_fc_port_template { struct nvme_fc_remote_port *,void *hw_queue_handle, struct nvmefc_fcp_req *); +int (*xmt_ls_rsp)(struct nvme_fc_local_port *localport, +struct nvme_fc_remote_port *rport, +struct nvmefc_ls_rsp *ls_rsp);u32 max_hw_queues;u16 max_sgl_segments;@@ -445,6 +537,34 @@ void nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport);int nvme_fc_set_remoteport_devloss (struct nvme_fc_remote_port *remoteport,u32 dev_loss_tmo); +/* +* Routine called to pass a NVME-FC LS request, received by the lldd, +* to the nvme-fc transport. +* +* If the return value is zero: the LS was successfully accepted by the +* transport. +* If the return value is non-zero: the transport has not accepted the +* LS. The lldd should ABTS-LS the LS. +* +* Note: if the LLDD receives and ABTS for the LS prior to the transport +* calling the ops->xmt_ls_rsp() routine to transmit a response, the LLDD +* shall mark the LS as aborted, and when the xmt_ls_rsp() is called: the +* response shall not be transmit and the struct nvmefc_ls_rsp() done +* routine shall be called. The LLDD may transmit the ABTS response as +* soon as the LS was marked or can delay until the xmt_ls_rsp() call is +* made. +* Note: if an RCV LS was successfully posted to the transport and the +* remoteport is then unregistered before xmt_ls_rsp() was called for +* the lsrsp structure, the transport will still call xmt_ls_rsp() +* afterward to cleanup the outstanding lsrsp structure. The LLDD should +* noop the transmission of the rsp and call the lsrsp->done() routine +* to allow the lsrsp structure to be released. +*/ +int nvme_fc_rcv_ls_req (struct nvme_fc_remote_port *remoteport, +struct nvmefc_ls_rsp *lsrsp, +void *lsreqbuf,u32 lsreqbuf_len); + +/* * *************** LLDD FC-NVME Target/Subsystem API *************** @@ -474,55 +594,6 @@ struct nvmet_fc_port_info { }; -/** -* struct nvmefc_tgt_ls_req - Structure used between LLDD and NVMET-FC -* layer to represent the exchange context for -* a FC-NVME Link Service (LS). -* -* The structure is allocated by the LLDD whenever a LS Request is received -* from the FC link. The address of the structure is passed to the nvmet-fc -* layer via the nvmet_fc_rcv_ls_req() call. The address of the structure -* will be passed back to the LLDD when the response is to be transmit. -* The LLDD is to use the address to map back to the LLDD exchange structure -* which maintains information such as the targetport the LS was received -* on, the remote FC NVME initiator that sent the LS, and any FC exchange -* context. Upon completion of the LS response transmit, the address of the -* structure will be passed back to the LS rsp done() routine, allowing the -* nvmet-fc layer to release dma resources. Upon completion of the done() -* routine, no further access will be made by the nvmet-fc layer and the -* LLDD can de-allocate the structure. -* -* Field initialization: -* At the time of the nvmet_fc_rcv_ls_req() call, there is no content that -* is valid in the structure. -* -* When the structure is used for the LLDD->xmt_ls_rsp() call, the nvmet-fc -* layer will fully set the fields in order to specify the response -* payload buffer and its length as well as the done routine to be called -* upon compeletion of the transmit. The nvmet-fc layer will also set a -* private pointer for its own use in the done routine. -* -* Values set by the NVMET-FC layer prior to calling the LLDD xmt_ls_rsp -* entrypoint. -* @rspbuf: pointer to the LS response buffer -* @rspdma: PCI DMA address of the LS response buffer -* @rsplen: Length, in bytes, of the LS response buffer -* @done: The callback routine the LLDD is to invoke upon completion of -* transmitting the LS response. req argument is the pointer to -* the original ls request. -* @nvmet_fc_private: pointer to an internal NVMET-FC layer structure used -* as part of the NVMET-FC processing. The LLDD is not to access -* this pointer. -*/ -struct nvmefc_tgt_ls_req { -void *rspbuf; -dma_addr_t rspdma; -u16 rsplen; - -void (*done)(struct nvmefc_tgt_ls_req *req); -void *nvmet_fc_private;/* LLDD is not to access !! */ -}; -/* Operations that NVME-FC layer may request the LLDD to perform for FCP */ enum {NVMET_FCOP_READDATA =1 ,/* xmt data to initiator */ @@ -697,17 +768,19 @@ struct nvmet_fc_target_port {* Entrypoint is Mandatory. * * @xmt_ls_rsp: Called to transmit the response to a FC-NVME FC-4 LS service. -* The nvmefc_tgt_ls_req structure is the same LLDD-supplied exchange +* The nvmefc_ls_rsp structure is the same LLDD-supplied exchange * structure specified in the nvmet_fc_rcv_ls_req() call made when -* the LS request was received. The structure will fully describe +* the LS request was received. The structure will fully describe * the buffers for the response payload and the dma address of the -* payload. The LLDD is to transmit the response (or return a non-zero -* errno status), and upon completion of the transmit, call the -* "done" routine specified in the nvmefc_tgt_ls_req structure -* (argument to done is the ls reqwuest structure itself). -* After calling the done routine, the LLDD shall consider the -* LS handling complete and the nvmefc_tgt_ls_req structure may -* be freed/released. +* payload. The LLDD is to transmit the response (or return a +* non-zero errno status), and upon completion of the transmit, call +* the "done" routine specified in the nvmefc_ls_rsp structure +* (argument to done is the address of the nvmefc_ls_rsp structure +* itself). Upon the completion of the done() routine, the LLDD shall +* consider the LS handling complete and the nvmefc_ls_rsp structure +* may be freed/released. +* The transport will always call the xmt_ls_rsp() routine for any +* LS received. * Entrypoint is Mandatory. * * @fcp_op: Called to perform a data transfer or transmit a response. @@ -802,6 +875,39 @@ struct nvmet_fc_target_port {* should cause the initiator to rescan the discovery controller * on the targetport. * +* @ls_req: Called to issue a FC-NVME FC-4 LS service request. +* The nvme_fc_ls_req structure will fully describe the buffers for +* the request payload and where to place the response payload. +* The targetport that is to issue the LS request is identified by +* the targetport argument. The remote port that is to receive the +* LS request is identified by the hosthandle argument. The nvmet-fc +* transport is only allowed to issue FC-NVME LS's on behalf of an +* association that was created prior by a Create Association LS. +* The hosthandle will originate from the LLDD in the struct +* nvmefc_ls_rsp structure for the Create Association LS that +* was delivered to the transport. The transport will save the +* hosthandle as an attribute of the association. If the LLDD +* loses connectivity with the remote port, it must call the +* nvmet_fc_invalidate_host() routine to remove any references to +* the remote port in the transport. +* The LLDD is to allocate an exchange, issue the LS request, obtain +* the LS response, and call the "done" routine specified in the +* request structure (argument to done is the ls request structure +* itself). +* Entrypoint is Optional - but highly recommended. +* +* @ls_abort: called to request the LLDD to abort the indicated ls request. +* The call may return before the abort has completed. After aborting +* the request, the LLDD must still call the ls request done routine +* indicating an FC transport Aborted status. +* Entrypoint is Mandatory if the ls_req entry point is specified. +* +* @host_release: called to inform the LLDD that the request to invalidate +* the host port indicated by the hosthandle has been fully completed. +* No associations exist with the host port and there will be no +* further references to hosthandle. +* Entrypoint is Mandatory if the lldd calls nvmet_fc_invalidate_host(). +* * @max_hw_queues: indicates the maximum number of hw queues the LLDD * supports for cpu affinitization. * Value is Mandatory. Must be at least 1. @@ -830,11 +936,19 @@ struct nvmet_fc_target_port {* area solely for the of the LLDD and its location is specified by * the targetport->private pointer. * Value is Mandatory. Allowed to be zero. +* +* @lsrqst_priv_sz: The LLDD sets this field to the amount of additional +* memory that it would like nvmet-fc layer to allocate on the LLDD's +* behalf whenever a ls request structure is allocated. The additional +* memory area is solely for use by the LLDD and its location is +* specified by the ls_request->private pointer. +* Value is Mandatory. Allowed to be zero. +* */ struct nvmet_fc_target_template {void (*targetport_delete)(struct nvmet_fc_target_port *tgtport);int (*xmt_ls_rsp)(struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *tls_req); +struct nvmefc_ls_rsp *ls_rsp);int (*fcp_op)(struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq); void (*fcp_abort)(struct nvmet_fc_target_port *tgtport,@@ -844,6 +958,11 @@ struct nvmet_fc_target_template {void (*defer_rcv)(struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq);void (*discovery_event)(struct nvmet_fc_target_port *tgtport); +int (*ls_req)(struct nvmet_fc_target_port *targetport, +void *hosthandle,struct nvmefc_ls_req *lsreq); +void (*ls_abort)(struct nvmet_fc_target_port *targetport, +void *hosthandle,struct nvmefc_ls_req *lsreq); +void (*host_release)(void *hosthandle);u32 max_hw_queues;u16 max_sgl_segments;@@ -852,7 +971,9 @@ struct nvmet_fc_target_template {u32 target_features; +/* sizes of additional private data for data structures */ u32 target_priv_sz; +u32 lsrqst_priv_sz; };@@ -863,10 +984,61 @@ int nvmet_fc_register_targetport(struct nvmet_fc_port_info *portinfo,int nvmet_fc_unregister_targetport (struct nvmet_fc_target_port *tgtport); +/* +* Routine called to pass a NVME-FC LS request, received by the lldd, +* to the nvmet-fc transport. +* +* If the return value is zero: the LS was successfully accepted by the +* transport. +* If the return value is non-zero: the transport has not accepted the +* LS. The lldd should ABTS-LS the LS. +* +* Note: if the LLDD receives and ABTS for the LS prior to the transport +* calling the ops->xmt_ls_rsp() routine to transmit a response, the LLDD +* shall mark the LS as aborted, and when the xmt_ls_rsp() is called: the +* response shall not be transmit and the struct nvmefc_ls_rsp() done +* routine shall be called. The LLDD may transmit the ABTS response as +* soon as the LS was marked or can delay until the xmt_ls_rsp() call is +* made. +* Note: if an RCV LS was successfully posted to the transport and the +* targetport is then unregistered before xmt_ls_rsp() was called for +* the lsrsp structure, the transport will still call xmt_ls_rsp() +* afterward to cleanup the outstanding lsrsp structure. The LLDD should +* noop the transmission of the rsp and call the lsrsp->done() routine +* to allow the lsrsp structure to be released. +*/ int nvmet_fc_rcv_ls_req (struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *lsreq, +void *hosthandle, +struct nvmefc_ls_rsp *rsp,void *lsreqbuf,u32 lsreqbuf_len); +/* +* Routine called by the LLDD whenever it has a logout or loss of +* connectivity to a NVME-FC host port which there had been active +* NVMe controllers for. The host port is indicated by the +* hosthandle. The hosthandle is given to the nvmet-fc transport +* when a NVME LS was received, typically to create a new association. +* The nvmet-fc transport will cache the hostport value with the +* association for use in LS requests for the association. +* When the LLDD calls this routine, the nvmet-fc transport will +* immediately terminate all associations that were created with +* the hosthandle host port. +* The LLDD, after calling this routine and having control returned, +* must assume the transport may subsequently utilize hosthandle as +* part of sending LS's to terminate the association. The LLDD +* should reject the LS's if they are attempted. +* Once the last association has terminated for the hosthandle host +* port, the nvmet-fc transport will call the ops->host_release() +* callback. As of the callback, the nvmet-fc transport will no +* longer reference hosthandle. +*/
include/linux/nvme-fc-driver.h int nvmet_fc_register_targetport(struct nvmet_fc_port_info *portinfo, · line 1044
Quoted 10 lines
include/linux/nvme-fc-driver.h +7
-0+void nvmet_fc_invalidate_host (struct nvmet_fc_target_port *tgtport, +void *hosthandle); + +/* +* If nvmet_fc_rcv_fcp_req returns non-zero, the transport has not accepted +* the FCP cmd. The lldd should ABTS-LS the cmd. +*/ int nvmet_fc_rcv_fcp_req (struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq,void *cmdiubuf,u32 cmdiubuf_len);
Looks Good.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 4/29 nvme-fc nvmet_fc nvme_fcloop: adapt code to changed names in api header
In reply to James Smartdeal with following naming changes in the header:
nvmefc_tgt_ls_req -> nvmefc_ls_rsp nvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_private
Change calling sequence to nvmet_fc_rcv_ls_req() for hosthandle.
Add stubs for new interfaces:host/fc.c: nvme_fc_rcv_ls_req()target/fc.c: nvmet_fc_invalidate_host()
--- drivers/nvme/host/fc.c | 35 ++++++++++++++++++++ drivers/nvme/target/fc.c | 77 ++++++++++++++++++++++++++++++++------------ drivers/nvme/target/fcloop.c | 20 ++++++------ 3 files changed, 102 insertions(+), 30 deletions(-)
drivers/nvme/host/fc.c +35 -0
@@ -1465,6 +1465,41 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)kfree (lsop); } +/** +* nvme_fc_rcv_ls_req - transport entry point called by an LLDD +* upon the reception of a NVME LS request. +* +* The nvme-fc layer will copy payload to an internal structure for +* processing. As such, upon completion of the routine, the LLDD may +* immediately free/reuse the LS request buffer passed in the call. +* +* If this routine returns error, the LLDD should abort the exchange. +* +* @remoteport: pointer to the (registered) remote port that the LS +* was received from. The remoteport is associated with +* a specific localport. +* @lsrsp: pointer to a nvmefc_ls_rsp response structure to be +* used to reference the exchange corresponding to the LS +* when issuing an ls response. +* @lsreqbuf: pointer to the buffer containing the LS Request +* @lsreqbuf_len: length, in bytes, of the received LS request +*/ +int +nvme_fc_rcv_ls_req (struct nvme_fc_remote_port *portptr, +struct nvmefc_ls_rsp *lsrsp, +void *lsreqbuf,u32 lsreqbuf_len) +{ +struct nvme_fc_rport *rport =remoteport_to_rport (portptr); +struct nvme_fc_lport *lport = rport->lport; + +/* validate there's a routine to transmit a response */ +if (!lport->ops->xmt_ls_rsp) +return (-EINVAL ); + +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_fc_rcv_ls_req); +/* *********************** NVME Ctrl Routines **************************** */
drivers/nvme/target/fc.c +57 -20
170 lines not on this page · raw message
drivers/nvme/target/fcloop.c +10 -10
@@ -227,7 +227,7 @@ struct fcloop_lsreq {struct fcloop_tport *tport;struct nvmefc_ls_req *lsreq;struct work_struct work; -struct nvmefc_tgt_ls_req tgt_ls_req; +struct nvmefc_ls_rsp ls_rsp;int status; };
@@ -265,9 +265,9 @@ struct fcloop_ini_fcpreq { };static inline struct fcloop_lsreq * -tgt_ls_req_to_lsreq (struct nvmefc_tgt_ls_req *tgt_lsreq) +ls_rsp_to_lsreq (struct nvmefc_ls_rsp *lsrsp) { - return container_of(tgt_lsreq, struct fcloop_lsreq, tgt_ls_req); + return container_of(lsrsp, struct fcloop_lsreq, ls_rsp); } static inline struct fcloop_fcpreq *
@@ -330,7 +330,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, tls_req->status =0 ; tls_req->tport = rport->targetport->private; - ret =nvmet_fc_rcv_ls_req (rport->targetport, &tls_req->tgt_ls_req, + ret =nvmet_fc_rcv_ls_req (rport->targetport,NULL , &tls_req->ls_rsp, lsreq->rqstaddr, lsreq->rqstlen);return ret;
@@ -338,15 +338,15 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,static int fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *tport, -struct nvmefc_tgt_ls_req *tgt_lsreq) +struct nvmefc_ls_rsp *lsrsp) { -struct fcloop_lsreq *tls_req =tgt_ls_req_to_lsreq (tgt_lsreq); +struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);struct nvmefc_ls_req *lsreq = tls_req->lsreq; -memcpy (lsreq->rspaddr, tgt_lsreq->rspbuf, - ((lsreq->rsplen < tgt_lsreq->rsplen) ? - lsreq->rsplen : tgt_lsreq->rsplen)); - tgt_lsreq->done (tgt_lsreq); +memcpy (lsreq->rspaddr, lsrsp->rspbuf, + ((lsreq->rsplen < lsrsp->rsplen) ? + lsreq->rsplen : lsrsp->rsplen)); + lsrsp->done (lsrsp);schedule_work (&tls_req->work);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 04/29] nvme-fc nvmet_fc nvme_fcloop: adapt code to changed names in api header
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 04/29] nvme-fc nvmet_fc nvme_fcloop: adapt code to changed names in api header
In reply to James SmartQuoted 10 lines · James Smart
deal with following naming changes in the header:
nvmefc_tgt_ls_req->nvmefc_ls_rspnvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_privateChange calling sequence to
nvmet_fc_rcv_ls_req()for hosthandle.Add stubs for new interfaces:
host/fc.c:nvme_fc_rcv_ls_req()target/fc.c:nvmet_fc_invalidate_host()
Quoted 7 lines
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 35 ++++++++++++++++++++ drivers/nvme/target/fc.c | 77 ++++++++++++++++++++++++++++++++------------ drivers/nvme/target/fcloop.c | 20 ++++++------ 3 files changed, 102 insertions(+), 30 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 04/29] nvme-fc nvmet_fc nvme_fcloop: adapt code to changed names in api header
In reply to James SmartQuoted 298 lines · James Smart
deal with following naming changes in the header:
nvmefc_tgt_ls_req -> nvmefc_ls_rsp nvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_privateChange calling sequence to
nvmet_fc_rcv_ls_req()for hosthandle.Add stubs for new interfaces:
host/fc.c:nvme_fc_rcv_ls_req()target/fc.c:nvmet_fc_invalidate_host()Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 35 ++++++++++++++++++++ drivers/nvme/target/fc.c | 77 ++++++++++++++++++++++++++++++++------------ drivers/nvme/target/fcloop.c | 20 ++++++------ 3 files changed, 102 insertions(+), 30 deletions(-)drivers/nvme/host/fc.c +35
-0@@ -1465,6 +1465,41 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)kfree (lsop); } +/** +* nvme_fc_rcv_ls_req - transport entry point called by an LLDD +* upon the reception of a NVME LS request. +* +* The nvme-fc layer will copy payload to an internal structure for +* processing. As such, upon completion of the routine, the LLDD may +* immediately free/reuse the LS request buffer passed in the call. +* +* If this routine returns error, the LLDD should abort the exchange. +* +* @remoteport: pointer to the (registered) remote port that the LS +* was received from. The remoteport is associated with +* a specific localport. +* @lsrsp: pointer to a nvmefc_ls_rsp response structure to be +* used to reference the exchange corresponding to the LS +* when issuing an ls response. +* @lsreqbuf: pointer to the buffer containing the LS Request +* @lsreqbuf_len: length, in bytes, of the received LS request +*/ +int +nvme_fc_rcv_ls_req (struct nvme_fc_remote_port *portptr, +struct nvmefc_ls_rsp *lsrsp, +void *lsreqbuf,u32 lsreqbuf_len) +{ +struct nvme_fc_rport *rport =remoteport_to_rport (portptr); +struct nvme_fc_lport *lport = rport->lport; + +/* validate there's a routine to transmit a response */ +if (!lport->ops->xmt_ls_rsp) +return (-EINVAL ); + +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_fc_rcv_ls_req); +/* *********************** NVME Ctrl Routines **************************** */ drivers/nvme/target/fc.c +57
-20@@ -28,7 +28,7 @@ struct nvmet_fc_tgtport;struct nvmet_fc_tgt_assoc ;struct nvmet_fc_ls_iod { -struct nvmefc_tgt_ls_req *lsreq; +struct nvmefc_ls_rsp *lsrsp;struct nvmefc_tgt_fcp_req *fcpreq;/* only if RS */ struct list_head ls_list;/* tgtport->ls_list */ @@ -1146,6 +1146,42 @@ __nvmet_fc_free_assocs(struct nvmet_fc_tgtport *tgtport)spin_unlock_irqrestore (&tgtport->lock, flags); } +/** +* nvmet_fc_invalidate_host - transport entry point called by an LLDD +* to remove references to a hosthandle for LS's. +* +* The nvmet-fc layer ensures that any references to the hosthandle +* on the targetport are forgotten (set to NULL). The LLDD will +* typically call this when a login with a remote host port has been +* lost, thus LS's for the remote host port are no longer possible. +* +* If an LS request is outstanding to the targetport/hosthandle (or +* issued concurrently with the call to invalidate the host), the +* LLDD is responsible for terminating/aborting the LS and completing +* the LS request. It is recommended that these terminations/aborts +* occur after calling to invalidate the host handle to avoid additional +* retries by the nvmet-fc transport. The nvmet-fc transport may +* continue to reference host handle while it cleans up outstanding +* NVME associations. The nvmet-fc transport will call the +* ops->host_release() callback to notify the LLDD that all references +* are complete and the related host handle can be recovered. +* Note: if there are no references, the callback may be called before +* the invalidate host call returns. +* +* @target_port: pointer to the (registered) target port that a prior +* LS was received on and which supplied the transport the +* hosthandle. +* @hosthandle: the handle (pointer) that represents the host port +* that no longer has connectivity and that LS's should +* no longer be directed to. +*/ +void +nvmet_fc_invalidate_host (struct nvmet_fc_target_port *target_port, +void *hosthandle) +{ +} +EXPORT_SYMBOL_GPL (nvmet_fc_invalidate_host); +/* * nvmet layer has called to terminate an association */ @@ -1371,7 +1407,7 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Create Association LS failed: %s\n" , validation_errors[ret]); - iod->lsreq->rsplen =nvmet_fc_format_rjt (acc, + iod->lsrsp->rsplen =nvmet_fc_format_rjt (acc,NVME_FC_MAX_LS_BUFFER_SIZE , rqst->w0.ls_cmd,FCNVME_RJT_RC_LOGIC ,FCNVME_RJT_EXP_NONE ,0 );@@ -1384,7 +1420,7 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport,/* format a response */ - iod->lsreq->rsplen =sizeof (*acc); + iod->lsrsp->rsplen =sizeof (*acc); nvmet_fc_format_rsp_hdr(acc, FCNVME_LS_ACC, fcnvme_lsdesc_len(@@ -1462,7 +1498,7 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Create Connection LS failed: %s\n" , validation_errors[ret]); - iod->lsreq->rsplen = nvmet_fc_format_rjt(acc, + iod->lsrsp->rsplen = nvmet_fc_format_rjt(acc, NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd, (ret == VERR_NO_ASSOC) ? FCNVME_RJT_RC_INV_ASSOC :@@ -1477,7 +1513,7 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport,/* format a response */ - iod->lsreq->rsplen =sizeof (*acc); + iod->lsrsp->rsplen =sizeof (*acc); nvmet_fc_format_rsp_hdr(acc, FCNVME_LS_ACC, fcnvme_lsdesc_len(sizeof(struct fcnvme_ls_cr_conn_acc)),@@ -1542,7 +1578,7 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Disconnect LS failed: %s\n" , validation_errors[ret]); - iod->lsreq->rsplen = nvmet_fc_format_rjt(acc, + iod->lsrsp->rsplen = nvmet_fc_format_rjt(acc, NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd, (ret == VERR_NO_ASSOC) ? FCNVME_RJT_RC_INV_ASSOC :@@ -1555,7 +1591,7 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,/* format a response */ - iod->lsreq->rsplen =sizeof (*acc); + iod->lsrsp->rsplen =sizeof (*acc); nvmet_fc_format_rsp_hdr(acc, FCNVME_LS_ACC, fcnvme_lsdesc_len(@@ -1577,9 +1613,9 @@ static void nvmet_fc_fcp_nvme_cmd_done(struct nvmet_req *nvme_req);static const struct nvmet_fabrics_ops nvmet_fc_tgt_fcp_ops;static void -nvmet_fc_xmt_ls_rsp_done (struct nvmefc_tgt_ls_req *lsreq) +nvmet_fc_xmt_ls_rsp_done (struct nvmefc_ls_rsp *lsrsp) { -struct nvmet_fc_ls_iod *iod = lsreq->nvmet_fc_private; +struct nvmet_fc_ls_iod *iod = lsrsp->nvme_fc_private;struct nvmet_fc_tgtport *tgtport = iod->tgtport; fc_dma_sync_single_for_cpu(tgtport->dev, iod->rspdma,@@ -1597,9 +1633,9 @@ nvmet_fc_xmt_ls_rsp(struct nvmet_fc_tgtport *tgtport,fc_dma_sync_single_for_device (tgtport->dev, iod->rspdma,NVME_FC_MAX_LS_BUFFER_SIZE ,DMA_TO_DEVICE ); - ret = tgtport->ops->xmt_ls_rsp (&tgtport->fc_target_port, iod->lsreq); + ret = tgtport->ops->xmt_ls_rsp (&tgtport->fc_target_port, iod->lsrsp);if (ret) -nvmet_fc_xmt_ls_rsp_done (iod->lsreq); +nvmet_fc_xmt_ls_rsp_done (iod->lsrsp); }/* @@ -1612,12 +1648,12 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport,struct fcnvme_ls_rqst_w0 *w0 = (struct fcnvme_ls_rqst_w0 *)iod->rqstbuf; - iod->lsreq->nvmet_fc_private = iod; - iod->lsreq->rspbuf = iod->rspbuf; - iod->lsreq->rspdma = iod->rspdma; - iod->lsreq->done = nvmet_fc_xmt_ls_rsp_done; + iod->lsrsp->nvme_fc_private = iod; + iod->lsrsp->rspbuf = iod->rspbuf; + iod->lsrsp->rspdma = iod->rspdma; + iod->lsrsp->done = nvmet_fc_xmt_ls_rsp_done;/* Be preventative. handlers will later set to valid length */ - iod->lsreq->rsplen =0 ; + iod->lsrsp->rsplen =0 ; iod->assoc =NULL ;@@ -1640,7 +1676,7 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport,nvmet_fc_ls_disconnect (tgtport, iod);break ;default : - iod->lsreq->rsplen =nvmet_fc_format_rjt (iod->rspbuf, + iod->lsrsp->rsplen =nvmet_fc_format_rjt (iod->rspbuf,NVME_FC_MAX_LS_BUFFER_SIZE , w0->ls_cmd,FCNVME_RJT_RC_INVAL ,FCNVME_RJT_EXP_NONE ,0 ); }@@ -1674,14 +1710,15 @@ nvmet_fc_handle_ls_rqst_work(struct work_struct *work)* * @target_port: pointer to the (registered) target port the LS was * received on. -* @lsreq: pointer to a lsreq request structure to be used to reference +* @lsrsp: pointer to a lsrsp structure to be used to reference * the exchange corresponding to the LS. * @lsreqbuf: pointer to the buffer containing the LS Request * @lsreqbuf_len: length, in bytes, of the received LS request */ int nvmet_fc_rcv_ls_req (struct nvmet_fc_target_port *target_port, -struct nvmefc_tgt_ls_req *lsreq, +void *hosthandle, +struct nvmefc_ls_rsp *lsrsp,void *lsreqbuf,u32 lsreqbuf_len) {struct nvmet_fc_tgtport *tgtport =targetport_to_tgtport (target_port);@@ -1699,7 +1736,7 @@ nvmet_fc_rcv_ls_req(struct nvmet_fc_target_port *target_port,return -ENOENT ; } - iod->lsreq = lsreq; + iod->lsrsp = lsrsp; iod->fcpreq =NULL ;memcpy (iod->rqstbuf, lsreqbuf, lsreqbuf_len); iod->rqstdatalen = lsreqbuf_len;drivers/nvme/target/fcloop.c +6
-7@@ -227,7 +227,7 @@ struct fcloop_lsreq {struct fcloop_tport *tport;struct nvmefc_ls_req *lsreq;struct work_struct work; -struct nvmefc_tgt_ls_req tgt_ls_req; +struct nvmefc_ls_rsp ls_rsp;int status; };@@ -265,9 +265,9 @@ struct fcloop_ini_fcpreq { };static inline struct fcloop_lsreq * -tgt_ls_req_to_lsreq (struct nvmefc_tgt_ls_req *tgt_lsreq) +ls_rsp_to_lsreq (struct nvmefc_ls_rsp *lsrsp) { - return container_of(tgt_lsreq, struct fcloop_lsreq, tgt_ls_req); + return container_of(lsrsp, struct fcloop_lsreq, ls_rsp); } static inline struct fcloop_fcpreq *@@ -330,7 +330,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, tls_req->status =0 ; tls_req->tport = rport->targetport->private; - ret =nvmet_fc_rcv_ls_req (rport->targetport, &tls_req->tgt_ls_req, + ret =nvmet_fc_rcv_ls_req (rport->targetport,NULL , &tls_req->ls_rsp, lsreq->rqstaddr, lsreq->rqstlen);return ret;@@ -338,15 +338,15 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,static int fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *tport, -struct nvmefc_tgt_ls_req *tgt_lsreq) +struct nvmefc_ls_rsp *lsrsp) { -struct fcloop_lsreq *tls_req =tgt_ls_req_to_lsreq (tgt_lsreq); +struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);struct nvmefc_ls_req *lsreq = tls_req->lsreq; - memcpy(lsreq->rspaddr, tgt_lsreq->rspbuf,
drivers/nvme/target/fcloop.c fcloop_ls_req(struct nvme_fc_local_port *localport, · line 352
Quoted 10 lines
drivers/nvme/target/fcloop.c +4
-3- ((lsreq->rsplen < tgt_lsreq->rsplen) ? - lsreq->rsplen : tgt_lsreq->rsplen)); - tgt_lsreq->done (tgt_lsreq); +memcpy (lsreq->rspaddr, lsrsp->rspbuf, + ((lsreq->rsplen < lsrsp->rsplen) ? + lsreq->rsplen : lsrsp->rsplen)); + lsrsp->done (lsrsp);schedule_work (&tls_req->work);
Looks Good.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 5/29 lpfc: adapt code to changed names in api header
In reply to James Smartdeal with following naming changes in the header:
nvmefc_tgt_ls_req -> nvmefc_ls_rsp nvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_private
Change calling sequence to nvmet_fc_rcv_ls_req() for hosthandle.
--- drivers/scsi/lpfc/lpfc_nvmet.c | 10 +++++----- drivers/scsi/lpfc/lpfc_nvmet.h | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)
drivers/scsi/lpfc/lpfc_nvmet.c +5 -5
@@ -302,7 +302,7 @@ lpfc_nvmet_xmt_ls_rsp_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe, struct lpfc_wcqe_complete *wcqe) {struct lpfc_nvmet_tgtport *tgtp; -struct nvmefc_tgt_ls_req *rsp; +struct nvmefc_ls_rsp *rsp;struct lpfc_nvmet_rcv_ctx *ctxp;uint32_t status, result;
@@ -335,7 +335,7 @@ lpfc_nvmet_xmt_ls_rsp_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe, } out: - rsp = &ctxp->ctx.ls_req; + rsp = &ctxp->ctx.ls_rsp;lpfc_nvmeio_data (phba,"NVMET LS CMPL: xri x%x stat x%x result x%x\n" , ctxp->oxid, status, result);
@@ -830,10 +830,10 @@ lpfc_nvmet_xmt_fcp_op_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe,static int lpfc_nvmet_xmt_ls_rsp (struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *rsp) +struct nvmefc_ls_rsp *rsp) {struct lpfc_nvmet_rcv_ctx *ctxp = - container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.ls_req); + container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.ls_rsp);struct lpfc_hba *phba = ctxp->phba;struct hbq_dmabuf *nvmebuf = (struct hbq_dmabuf *)ctxp->rqb_buffer;
@@ -2000,7 +2000,7 @@ lpfc_nvmet_unsol_ls_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring,* lpfc_nvmet_xmt_ls_rsp_cmp should free the allocated ctxp. */ atomic_inc (&tgtp->rcv_ls_req_in); - rc =nvmet_fc_rcv_ls_req (phba->targetport, &ctxp->ctx.ls_req, + rc =nvmet_fc_rcv_ls_req (phba->targetport,NULL , &ctxp->ctx.ls_rsp, payload, size); lpfc_printf_log(phba, KERN_INFO, LOG_NVME_DISC,
drivers/scsi/lpfc/lpfc_nvmet.h +1 -1
@@ -105,7 +105,7 @@ struct lpfc_nvmet_ctx_info {struct lpfc_nvmet_rcv_ctx {union { -struct nvmefc_tgt_ls_req ls_req; +struct nvmefc_ls_rsp ls_rsp;struct nvmefc_tgt_fcp_req fcp_req; } ctx;struct list_head list;
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 05/29] lpfc: adapt code to changed names in api header
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 05/29] lpfc: adapt code to changed names in api header
In reply to James SmartQuoted 4 lines · James Smart
deal with following naming changes in the header:
nvmefc_tgt_ls_req->nvmefc_ls_rspnvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_private
Quoted 8 lines
Change calling sequence to
nvmet_fc_rcv_ls_req()for hosthandle.Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvmet.c | 10 +++++----- drivers/scsi/lpfc/lpfc_nvmet.h | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)Please merge this patch with the two previous ones; we should strive to
make every patch self-contained in the sense that it allows for a clean compilation.
Otherwise you'll break bisecting.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 05/29] lpfc: adapt code to changed names in api header
In reply to James SmartQuoted 62 lines · James Smart
deal with following naming changes in the header:
nvmefc_tgt_ls_req -> nvmefc_ls_rsp nvmefc_tgt_ls_req.nvmet_fc_private -> nvmefc_ls_rsp.nvme_fc_privateChange calling sequence to
nvmet_fc_rcv_ls_req()for hosthandle.Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvmet.c | 10 +++++----- drivers/scsi/lpfc/lpfc_nvmet.h | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)drivers/scsi/lpfc/lpfc_nvmet.c +5
-5@@ -302,7 +302,7 @@ lpfc_nvmet_xmt_ls_rsp_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe, struct lpfc_wcqe_complete *wcqe) {struct lpfc_nvmet_tgtport *tgtp; -struct nvmefc_tgt_ls_req *rsp; +struct nvmefc_ls_rsp *rsp;struct lpfc_nvmet_rcv_ctx *ctxp;uint32_t status, result;@@ -335,7 +335,7 @@ lpfc_nvmet_xmt_ls_rsp_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe, } out: - rsp = &ctxp->ctx.ls_req; + rsp = &ctxp->ctx.ls_rsp;lpfc_nvmeio_data (phba,"NVMET LS CMPL: xri x%x stat x%x result x%x\n" , ctxp->oxid, status, result);@@ -830,10 +830,10 @@ lpfc_nvmet_xmt_fcp_op_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe,static int lpfc_nvmet_xmt_ls_rsp (struct nvmet_fc_target_port *tgtport, -struct nvmefc_tgt_ls_req *rsp) +struct nvmefc_ls_rsp *rsp) {struct lpfc_nvmet_rcv_ctx *ctxp = - container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.ls_req); + container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.ls_rsp);struct lpfc_hba *phba = ctxp->phba;struct hbq_dmabuf *nvmebuf = (struct hbq_dmabuf *)ctxp->rqb_buffer;@@ -2000,7 +2000,7 @@ lpfc_nvmet_unsol_ls_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring,* lpfc_nvmet_xmt_ls_rsp_cmp should free the allocated ctxp. */ atomic_inc (&tgtp->rcv_ls_req_in); - rc =nvmet_fc_rcv_ls_req (phba->targetport, &ctxp->ctx.ls_req, + rc =nvmet_fc_rcv_ls_req (phba->targetport,NULL , &ctxp->ctx.ls_rsp, payload, size); lpfc_printf_log(phba, KERN_INFO, LOG_NVME_DISC,drivers/scsi/lpfc/lpfc_nvmet.h +0
-0
drivers/scsi/lpfc/lpfc_nvmet.h struct lpfc_nvmet_ctx_info { · line 111
Quoted 8 lines
drivers/scsi/lpfc/lpfc_nvmet.h +1
-1@@ -105,7 +105,7 @@ struct lpfc_nvmet_ctx_info {struct lpfc_nvmet_rcv_ctx {union { -struct nvmefc_tgt_ls_req ls_req; +struct nvmefc_ls_rsp ls_rsp;struct nvmefc_tgt_fcp_req fcp_req; } ctx;struct list_head list;
FWIW, Looks good.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 6/29 nvme-fcloop: Fix deallocation of working context
In reply to James SmartThere's been a longstanding bug of LS completions which freed ls op's, particularly the disconnect LS, while executing on a work context that is in the memory being free. Not a good thing to do.
Rework LS handling to make callbacks in the rport context rather than the ls_request context.
--- drivers/nvme/target/fcloop.c | 76 ++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 24 deletions(-)
drivers/nvme/target/fcloop.c +52 -24
@@ -198,10 +198,13 @@ struct fcloop_lport_priv { };struct fcloop_rport { -struct nvme_fc_remote_port *remoteport; -struct nvmet_fc_target_port *targetport; -struct fcloop_nport *nport; -struct fcloop_lport *lport; +struct nvme_fc_remote_port *remoteport; +struct nvmet_fc_target_port *targetport; +struct fcloop_nport *nport; +struct fcloop_lport *lport; +spinlock_t lock; +struct list_head ls_list; +struct work_struct ls_work; };struct fcloop_tport {
@@ -224,11 +227,10 @@ struct fcloop_nport { };struct fcloop_lsreq { -struct fcloop_tport *tport;struct nvmefc_ls_req *lsreq; -struct work_struct work;struct nvmefc_ls_rsp ls_rsp;int status; +struct list_head ls_list;/* fcloop_rport->ls_list */ };struct fcloop_rscn {
@@ -292,21 +294,32 @@ fcloop_delete_queue(struct nvme_fc_local_port *localport, { } - -/* -* Transmit of LS RSP done (e.g. buffers all set). call back up -* initiator "done" flows. -*/ static void -fcloop_tgt_lsrqst_done_work (struct work_struct *work) +fcloop_rport_lsrqst_work (struct work_struct *work) { -struct fcloop_lsreq *tls_req = - container_of(work, struct fcloop_lsreq, work); -struct fcloop_tport *tport = tls_req->tport; -struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_rport *rport = + container_of(work, struct fcloop_rport, ls_work); +struct fcloop_lsreq *tls_req; -if (!tport || tport->remoteport) - lsreq->done (lsreq, tls_req->status); +spin_lock (&rport->lock); +for (;;) { + tls_req =list_first_entry_or_null (&rport->ls_list, + struct fcloop_lsreq, ls_list); +if (!tls_req) +break ; + +list_del (&tls_req->ls_list); +spin_unlock (&rport->lock); + + tls_req->lsreq->done (tls_req->lsreq, tls_req->status); +/* +* callee may free memory containing tls_req. +* do not reference lsreq after this. +*/ + +spin_lock (&rport->lock); + } +spin_unlock (&rport->lock); } static int
@@ -319,17 +332,18 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,int ret =0 ; tls_req->lsreq = lsreq; -INIT_WORK (&tls_req->work, fcloop_tgt_lsrqst_done_work); +INIT_LIST_HEAD (&tls_req->ls_list);if (!rport->targetport) { tls_req->status = -ECONNREFUSED ; - tls_req->tport =NULL ; -schedule_work (&tls_req->work); +spin_lock (&rport->lock); +list_add_tail (&rport->ls_list, &tls_req->ls_list); +spin_unlock (&rport->lock); +schedule_work (&rport->ls_work);return ret; } tls_req->status =0 ; - tls_req->tport = rport->targetport->private; ret =nvmet_fc_rcv_ls_req (rport->targetport,NULL , &tls_req->ls_rsp, lsreq->rqstaddr, lsreq->rqstlen);
@@ -337,18 +351,28 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, }static int -fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *tport, +fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *targetport,struct nvmefc_ls_rsp *lsrsp) {struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_tport *tport = targetport->private; +struct nvme_fc_remote_port *remoteport = tport->remoteport; +struct fcloop_rport *rport;memcpy (lsreq->rspaddr, lsrsp->rspbuf, ((lsreq->rsplen < lsrsp->rsplen) ? lsreq->rsplen : lsrsp->rsplen)); + lsrsp->done (lsrsp); -schedule_work (&tls_req->work); +if (remoteport) { + rport = remoteport->private; +spin_lock (&rport->lock); +list_add_tail (&rport->ls_list, &tls_req->ls_list); +spin_unlock (&rport->lock); +schedule_work (&rport->ls_work); + }return 0 ; }
@@ -834,6 +858,7 @@ fcloop_remoteport_delete(struct nvme_fc_remote_port *remoteport) {struct fcloop_rport *rport = remoteport->private; +flush_work (&rport->ls_work);fcloop_nport_put (rport->nport); }
@@ -1136,6 +1161,9 @@ fcloop_create_remote_port(struct device *dev, struct device_attribute *attr, rport->nport = nport; rport->lport = nport->lport; nport->rport = rport; +spin_lock_init (&rport->lock); +INIT_WORK (&rport->ls_work, fcloop_rport_lsrqst_work); +INIT_LIST_HEAD (&rport->ls_list);return count; }
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 06/29] nvme-fcloop: Fix deallocation of working context
Is this directly related to the series? Or should this be a dedicated bug fix (that in turn will go to stable etc)?
Quoted 79 lines · James Smart
There's been a longstanding bug of LS completions which freed ls op's, particularly the disconnect LS, while executing on a work context that is in the memory being free. Not a good thing to do.
Rework LS handling to make callbacks in the rport context rather than the
ls_requestcontext.Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fcloop.c | 76 ++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 24 deletions(-)drivers/nvme/target/fcloop.c +21
-18@@ -198,10 +198,13 @@ struct fcloop_lport_priv { };struct fcloop_rport { -struct nvme_fc_remote_port *remoteport; -struct nvmet_fc_target_port *targetport; -struct fcloop_nport *nport; -struct fcloop_lport *lport; +struct nvme_fc_remote_port *remoteport; +struct nvmet_fc_target_port *targetport; +struct fcloop_nport *nport; +struct fcloop_lport *lport; +spinlock_t lock; +struct list_head ls_list; +struct work_struct ls_work; };struct fcloop_tport {@@ -224,11 +227,10 @@ struct fcloop_nport { };struct fcloop_lsreq { -struct fcloop_tport *tport;struct nvmefc_ls_req *lsreq; -struct work_struct work;struct nvmefc_ls_rsp ls_rsp;int status; +struct list_head ls_list;/* fcloop_rport->ls_list */ };struct fcloop_rscn {@@ -292,21 +294,32 @@ fcloop_delete_queue(struct nvme_fc_local_port *localport, { } - -/* -* Transmit of LS RSP done (e.g. buffers all set). call back up -* initiator "done" flows. -*/ static void -fcloop_tgt_lsrqst_done_work (struct work_struct *work) +fcloop_rport_lsrqst_work (struct work_struct *work) { -struct fcloop_lsreq *tls_req = - container_of(work, struct fcloop_lsreq, work); -struct fcloop_tport *tport = tls_req->tport; -struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_rport *rport = + container_of(work, struct fcloop_rport, ls_work); +struct fcloop_lsreq *tls_req; -if (!tport || tport->remoteport) - lsreq->done (lsreq, tls_req->status); +spin_lock (&rport->lock); +for (;;) { + tls_req =list_first_entry_or_null (&rport->ls_list, + struct fcloop_lsreq, ls_list); +if (!tls_req) +break ; + +list_del (&tls_req->ls_list); +spin_unlock (&rport->lock);
drivers/nvme/target/fcloop.c fcloop_delete_queue(struct nvme_fc_local_port *localport, · line 322
Quoted 10 lines
drivers/nvme/target/fcloop.c +10
-0+ + tls_req->lsreq->done (tls_req->lsreq, tls_req->status); +/* +* callee may free memory containing tls_req. +* do not reference lsreq after this. +*/ + +spin_lock (&rport->lock); + } +spin_unlock (&rport->lock);
Won't it be easier to splice to a local list instead?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 06/29] nvme-fcloop: Fix deallocation of working context
In reply to James SmartQuoted 4 lines · James Smart
There's been a longstanding bug of LS completions which freed ls op's, particularly the disconnect LS, while executing on a work context that is in the memory being free. Not a good thing to do.
Quoted 8 lines
Rework LS handling to make callbacks in the rport context rather than the
ls_requestcontext.Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fcloop.c | 76 ++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 24 deletions(-)
[ .. ]
As a nice side effect, this is the patch which fixes the crash with fcloop I've been seeing (and complained about) with my new fcloop blktest.
Consider sending this one separately.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 06/29] nvme-fcloop: Fix deallocation of working context
In reply to James SmartQuoted 156 lines · James Smart
There's been a longstanding bug of LS completions which freed ls op's, particularly the disconnect LS, while executing on a work context that is in the memory being free. Not a good thing to do.
Rework LS handling to make callbacks in the rport context rather than the
ls_requestcontext.Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fcloop.c | 76 ++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 24 deletions(-)drivers/nvme/target/fcloop.c +49
-24@@ -198,10 +198,13 @@ struct fcloop_lport_priv { };struct fcloop_rport { -struct nvme_fc_remote_port *remoteport; -struct nvmet_fc_target_port *targetport; -struct fcloop_nport *nport; -struct fcloop_lport *lport; +struct nvme_fc_remote_port *remoteport; +struct nvmet_fc_target_port *targetport; +struct fcloop_nport *nport; +struct fcloop_lport *lport; +spinlock_t lock; +struct list_head ls_list; +struct work_struct ls_work; };struct fcloop_tport {@@ -224,11 +227,10 @@ struct fcloop_nport { };struct fcloop_lsreq { -struct fcloop_tport *tport;struct nvmefc_ls_req *lsreq; -struct work_struct work;struct nvmefc_ls_rsp ls_rsp;int status; +struct list_head ls_list;/* fcloop_rport->ls_list */ };struct fcloop_rscn {@@ -292,21 +294,32 @@ fcloop_delete_queue(struct nvme_fc_local_port *localport, { } - -/* -* Transmit of LS RSP done (e.g. buffers all set). call back up -* initiator "done" flows. -*/ static void -fcloop_tgt_lsrqst_done_work (struct work_struct *work) +fcloop_rport_lsrqst_work (struct work_struct *work) { -struct fcloop_lsreq *tls_req = - container_of(work, struct fcloop_lsreq, work); -struct fcloop_tport *tport = tls_req->tport; -struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_rport *rport = + container_of(work, struct fcloop_rport, ls_work); +struct fcloop_lsreq *tls_req; -if (!tport || tport->remoteport) - lsreq->done (lsreq, tls_req->status); +spin_lock (&rport->lock); +for (;;) { + tls_req =list_first_entry_or_null (&rport->ls_list, + struct fcloop_lsreq, ls_list); +if (!tls_req) +break ; + +list_del (&tls_req->ls_list); +spin_unlock (&rport->lock); + + tls_req->lsreq->done (tls_req->lsreq, tls_req->status); +/* +* callee may free memory containing tls_req. +* do not reference lsreq after this. +*/ + +spin_lock (&rport->lock); + } +spin_unlock (&rport->lock); } static int@@ -319,17 +332,18 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,int ret =0 ; tls_req->lsreq = lsreq; -INIT_WORK (&tls_req->work, fcloop_tgt_lsrqst_done_work); +INIT_LIST_HEAD (&tls_req->ls_list);if (!rport->targetport) { tls_req->status = -ECONNREFUSED ; - tls_req->tport =NULL ; -schedule_work (&tls_req->work); +spin_lock (&rport->lock); +list_add_tail (&rport->ls_list, &tls_req->ls_list); +spin_unlock (&rport->lock); +schedule_work (&rport->ls_work);return ret; } tls_req->status =0 ; - tls_req->tport = rport->targetport->private; ret =nvmet_fc_rcv_ls_req (rport->targetport,NULL , &tls_req->ls_rsp, lsreq->rqstaddr, lsreq->rqstlen);@@ -337,18 +351,28 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, }static int -fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *tport, +fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *targetport,struct nvmefc_ls_rsp *lsrsp) {struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_tport *tport = targetport->private; +struct nvme_fc_remote_port *remoteport = tport->remoteport; +struct fcloop_rport *rport;memcpy (lsreq->rspaddr, lsrsp->rspbuf, ((lsreq->rsplen < lsrsp->rsplen) ? lsreq->rsplen : lsrsp->rsplen)); + lsrsp->done (lsrsp); -schedule_work (&tls_req->work); +if (remoteport) { + rport = remoteport->private; +spin_lock (&rport->lock); +list_add_tail (&rport->ls_list, &tls_req->ls_list); +spin_unlock (&rport->lock); +schedule_work (&rport->ls_work); + }return 0 ; }@@ -834,6 +858,7 @@ fcloop_remoteport_delete(struct nvme_fc_remote_port *remoteport) {struct fcloop_rport *rport = remoteport->private; +flush_work (&rport->ls_work);fcloop_nport_put (rport->nport); }
drivers/nvme/target/fcloop.c fcloop_create_remote_port(struct device *dev, struct device_attribute *attr, · line 1169
Quoted 9 lines
drivers/nvme/target/fcloop.c +3
-0@@ -1136,6 +1161,9 @@ fcloop_create_remote_port(struct device *dev, struct device_attribute *attr, rport->nport = nport; rport->lport = nport->lport; nport->rport = rport; +spin_lock_init (&rport->lock); +INIT_WORK (&rport->ls_work, fcloop_rport_lsrqst_work); +INIT_LIST_HEAD (&rport->ls_list);return count; }
Looks Good.
Signature · 6 lines
- Himanshu _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 7/29 nvme-fc nvmet-fc: refactor for common LS definitions
In reply to James SmartRoutines in the target will want to be used in the host as well. Error definitions should now shared as both sides will process requests and responses to requests.
Moved common declarations to new fc.h header kept in the host subdirectory.
--- drivers/nvme/host/fc.c | 36 +------------ drivers/nvme/host/fc.h | 133 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 115 ++++------------------------------------ 3 files changed, 143 insertions(+), 141 deletions(-) create mode 100644 drivers/nvme/host/fc.h
drivers/nvme/host/fc.c +1 -35
@@ -14,6 +14,7 @@#include "fabrics.h" #include <linux/nvme-fc-driver.h> #include <linux/nvme-fc.h> +#include "fc.h" #include <scsi/scsi_transport_fc.h> /* *************************** Data Structures/Defines ****************** */
@@ -1141,41 +1142,6 @@ nvme_fc_send_ls_req_async(struct nvme_fc_rport *rport,return __nvme_fc_send_ls_req (rport, lsop, done); } -/* Validation Error indexes into the string table below */ -enum { -VERR_NO_ERROR =0 , -VERR_LSACC =1 , -VERR_LSDESC_RQST =2 , -VERR_LSDESC_RQST_LEN =3 , -VERR_ASSOC_ID =4 , -VERR_ASSOC_ID_LEN =5 , -VERR_CONN_ID =6 , -VERR_CONN_ID_LEN =7 , -VERR_CR_ASSOC =8 , -VERR_CR_ASSOC_ACC_LEN =9 , -VERR_CR_CONN =10 , -VERR_CR_CONN_ACC_LEN =11 , -VERR_DISCONN =12 , -VERR_DISCONN_ACC_LEN =13 , -}; - -static char *validation_errors[] = { -"OK" , -"Not LS_ACC" , -"Not LSDESC_RQST" , -"Bad LSDESC_RQST Length" , -"Not Association ID" , -"Bad Association ID Length" , -"Not Connection ID" , -"Bad Connection ID Length" , -"Not CR_ASSOC Rqst" , -"Bad CR_ASSOC ACC Length" , -"Not CR_CONN Rqst" , -"Bad CR_CONN ACC Length" , -"Not Disconnect Rqst" , -"Bad Disconnect ACC Length" , -}; -static int nvme_fc_connect_admin_queue (struct nvme_fc_ctrl *ctrl,struct nvme_fc_queue *queue,u16 qsize,u16 ersp_ratio)
drivers/nvme/host/fc.h new file +133 -0
133 lines not on this page · raw message
drivers/nvme/target/fc.c +9 -106
172 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 07/29] nvme-fc nvmet-fc: refactor for common LS definitions
Quoted 20 lines · James Smart
Routines in the target will want to be used in the host as well. Error definitions should now shared as both sides will process requests and responses to requests.
Moved common declarations to new
fc.hheader kept in the host subdirectory.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 36 +------------ drivers/nvme/host/fc.h | 133 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 115 ++++------------------------------------ 3 files changed, 143 insertions(+), 141 deletions(-) create mode 100644 drivers/nvme/host/fc.hdrivers/nvme/host/fc.c +0
-0
drivers/nvme/host/fc.c line 20
Quoted 7 lines
drivers/nvme/host/fc.c +1
-0@@ -14,6 +14,7 @@#include "fabrics.h" #include <linux/nvme-fc-driver.h> #include <linux/nvme-fc.h> +#include "fc.h" #include <scsi/scsi_transport_fc.h> /* *************************** Data Structures/Defines ****************** */
This doesn't apply cleanly (the first line should read '#include "trace.h"), but other than that:
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 07/29] nvme-fc nvmet-fc: refactor for common LS definitions
In reply to James SmartQuoted 386 lines · James Smart
Routines in the target will want to be used in the host as well. Error definitions should now shared as both sides will process requests and responses to requests.
Moved common declarations to new
fc.hheader kept in the host subdirectory.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 36 +------------ drivers/nvme/host/fc.h | 133 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 115 ++++------------------------------------ 3 files changed, 143 insertions(+), 141 deletions(-)create mode 100644
drivers/nvme/host/fc.hdrivers/nvme/host/fc.c +1
-35@@ -14,6 +14,7 @@#include "fabrics.h" #include <linux/nvme-fc-driver.h> #include <linux/nvme-fc.h> +#include "fc.h" #include <scsi/scsi_transport_fc.h> /* *************************** Data Structures/Defines ****************** */ @@ -1141,41 +1142,6 @@ nvme_fc_send_ls_req_async(struct nvme_fc_rport *rport,return __nvme_fc_send_ls_req (rport, lsop, done); } -/* Validation Error indexes into the string table below */ -enum { -VERR_NO_ERROR =0 , -VERR_LSACC =1 , -VERR_LSDESC_RQST =2 , -VERR_LSDESC_RQST_LEN =3 , -VERR_ASSOC_ID =4 , -VERR_ASSOC_ID_LEN =5 , -VERR_CONN_ID =6 , -VERR_CONN_ID_LEN =7 , -VERR_CR_ASSOC =8 , -VERR_CR_ASSOC_ACC_LEN =9 , -VERR_CR_CONN =10 , -VERR_CR_CONN_ACC_LEN =11 , -VERR_DISCONN =12 , -VERR_DISCONN_ACC_LEN =13 , -}; - -static char *validation_errors[] = { -"OK" , -"Not LS_ACC" , -"Not LSDESC_RQST" , -"Bad LSDESC_RQST Length" , -"Not Association ID" , -"Bad Association ID Length" , -"Not Connection ID" , -"Bad Connection ID Length" , -"Not CR_ASSOC Rqst" , -"Bad CR_ASSOC ACC Length" , -"Not CR_CONN Rqst" , -"Bad CR_CONN ACC Length" , -"Not Disconnect Rqst" , -"Bad Disconnect ACC Length" , -}; -static int nvme_fc_connect_admin_queue (struct nvme_fc_ctrl *ctrl,struct nvme_fc_queue *queue,u16 qsize,u16 ersp_ratio)drivers/nvme/host/fc.h new file +133
-0@@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2016, Avago Technologies +*/ + +#ifndef _NVME_FC_TRANSPORT_H +#define _NVME_FC_TRANSPORT_H 1 + + +/* +* Common definitions between the nvme_fc (host) transport and +* nvmet_fc (target) transport implementation. +*/ + +/* +* ****************** FC-NVME LS HANDLING ****************** +*/ + +static inline void +nvme_fc_format_rsp_hdr (void *buf,u8 ls_cmd,__be32 desc_len,u8 rqst_ls_cmd) +{ +struct fcnvme_ls_acc_hdr *acc = buf; + + acc->w0.ls_cmd = ls_cmd; + acc->desc_list_len = desc_len; + acc->rqst.desc_tag =cpu_to_be32 (FCNVME_LSDESC_RQST ); + acc->rqst.desc_len = +fcnvme_lsdesc_len (sizeof (struct fcnvme_lsdesc_rqst )); + acc->rqst.w0.ls_cmd = rqst_ls_cmd; +} + +static inline int +nvme_fc_format_rjt (void *buf,u16 buflen,u8 ls_cmd, +u8 reason,u8 explanation,u8 vendor) +{ +struct fcnvme_ls_rjt *rjt = buf; + +nvme_fc_format_rsp_hdr (buf,FCNVME_LSDESC_RQST , +fcnvme_lsdesc_len (sizeof (struct fcnvme_ls_rjt )), + ls_cmd); + rjt->rjt.desc_tag =cpu_to_be32 (FCNVME_LSDESC_RJT ); + rjt->rjt.desc_len =fcnvme_lsdesc_len (sizeof (struct fcnvme_lsdesc_rjt )); + rjt->rjt.reason_code = reason; + rjt->rjt.reason_explanation = explanation; + rjt->rjt.vendor = vendor; + +return sizeof (struct fcnvme_ls_rjt ); +} + +/* Validation Error indexes into the string table below */ +enum { +VERR_NO_ERROR =0 , +VERR_CR_ASSOC_LEN =1 , +VERR_CR_ASSOC_RQST_LEN =2 , +VERR_CR_ASSOC_CMD =3 , +VERR_CR_ASSOC_CMD_LEN =4 , +VERR_ERSP_RATIO =5 , +VERR_ASSOC_ALLOC_FAIL =6 , +VERR_QUEUE_ALLOC_FAIL =7 , +VERR_CR_CONN_LEN =8 , +VERR_CR_CONN_RQST_LEN =9 , +VERR_ASSOC_ID =10 , +VERR_ASSOC_ID_LEN =11 , +VERR_NO_ASSOC =12 , +VERR_CONN_ID =13 , +VERR_CONN_ID_LEN =14 , +VERR_INVAL_CONN =15 , +VERR_CR_CONN_CMD =16 , +VERR_CR_CONN_CMD_LEN =17 , +VERR_DISCONN_LEN =18 , +VERR_DISCONN_RQST_LEN =19 , +VERR_DISCONN_CMD =20 , +VERR_DISCONN_CMD_LEN =21 , +VERR_DISCONN_SCOPE =22 , +VERR_RS_LEN =23 , +VERR_RS_RQST_LEN =24 , +VERR_RS_CMD =25 , +VERR_RS_CMD_LEN =26 , +VERR_RS_RCTL =27 , +VERR_RS_RO =28 , +VERR_LSACC =29 , +VERR_LSDESC_RQST =30 , +VERR_LSDESC_RQST_LEN =31 , +VERR_CR_ASSOC =32 , +VERR_CR_ASSOC_ACC_LEN =33 , +VERR_CR_CONN =34 , +VERR_CR_CONN_ACC_LEN =35 , +VERR_DISCONN =36 , +VERR_DISCONN_ACC_LEN =37 , +}; + +static char *validation_errors[] = { +"OK" , +"Bad CR_ASSOC Length" , +"Bad CR_ASSOC Rqst Length" , +"Not CR_ASSOC Cmd" , +"Bad CR_ASSOC Cmd Length" , +"Bad Ersp Ratio" , +"Association Allocation Failed" , +"Queue Allocation Failed" , +"Bad CR_CONN Length" , +"Bad CR_CONN Rqst Length" , +"Not Association ID" , +"Bad Association ID Length" , +"No Association" , +"Not Connection ID" , +"Bad Connection ID Length" , +"Invalid Connection ID" , +"Not CR_CONN Cmd" , +"Bad CR_CONN Cmd Length" , +"Bad DISCONN Length" , +"Bad DISCONN Rqst Length" , +"Not DISCONN Cmd" , +"Bad DISCONN Cmd Length" , +"Bad Disconnect Scope" , +"Bad RS Length" , +"Bad RS Rqst Length" , +"Not RS Cmd" , +"Bad RS Cmd Length" , +"Bad RS R_CTL" , +"Bad RS Relative Offset" , +"Not LS_ACC" , +"Not LSDESC_RQST" , +"Bad LSDESC_RQST Length" , +"Not CR_ASSOC Rqst" , +"Bad CR_ASSOC ACC Length" , +"Not CR_CONN Rqst" , +"Bad CR_CONN ACC Length" , +"Not Disconnect Rqst" , +"Bad Disconnect ACC Length" , +}; + +#endif /* _NVME_FC_TRANSPORT_H */ drivers/nvme/target/fc.c +8
-105@@ -14,6 +14,7 @@#include "nvmet.h" #include <linux/nvme-fc-driver.h> #include <linux/nvme-fc.h> +#include "../host/fc.h" /* *************************** Data Structures/Defines ****************** */ @@ -1258,102 +1259,6 @@ EXPORT_SYMBOL_GPL(nvmet_fc_unregister_targetport);static void -nvmet_fc_format_rsp_hdr (void *buf,u8 ls_cmd,__be32 desc_len,u8 rqst_ls_cmd) -{ -struct fcnvme_ls_acc_hdr *acc = buf; - - acc->w0.ls_cmd = ls_cmd; - acc->desc_list_len = desc_len; - acc->rqst.desc_tag =cpu_to_be32 (FCNVME_LSDESC_RQST ); - acc->rqst.desc_len = -fcnvme_lsdesc_len (sizeof (struct fcnvme_lsdesc_rqst )); - acc->rqst.w0.ls_cmd = rqst_ls_cmd; -} - -static int -nvmet_fc_format_rjt (void *buf,u16 buflen,u8 ls_cmd, -u8 reason,u8 explanation,u8 vendor) -{ -struct fcnvme_ls_rjt *rjt = buf; - -nvmet_fc_format_rsp_hdr (buf,FCNVME_LSDESC_RQST , -fcnvme_lsdesc_len (sizeof (struct fcnvme_ls_rjt )), - ls_cmd); - rjt->rjt.desc_tag =cpu_to_be32 (FCNVME_LSDESC_RJT ); - rjt->rjt.desc_len =fcnvme_lsdesc_len (sizeof (struct fcnvme_lsdesc_rjt )); - rjt->rjt.reason_code = reason; - rjt->rjt.reason_explanation = explanation; - rjt->rjt.vendor = vendor; - -return sizeof (struct fcnvme_ls_rjt ); -} - -/* Validation Error indexes into the string table below */ -enum { -VERR_NO_ERROR =0 , -VERR_CR_ASSOC_LEN =1 , -VERR_CR_ASSOC_RQST_LEN =2 , -VERR_CR_ASSOC_CMD =3 , -VERR_CR_ASSOC_CMD_LEN =4 , -VERR_ERSP_RATIO =5 , -VERR_ASSOC_ALLOC_FAIL =6 , -VERR_QUEUE_ALLOC_FAIL =7 , -VERR_CR_CONN_LEN =8 , -VERR_CR_CONN_RQST_LEN =9 , -VERR_ASSOC_ID =10 , -VERR_ASSOC_ID_LEN =11 , -VERR_NO_ASSOC =12 , -VERR_CONN_ID =13 , -VERR_CONN_ID_LEN =14 , -VERR_NO_CONN =15 , -VERR_CR_CONN_CMD =16 , -VERR_CR_CONN_CMD_LEN =17 , -VERR_DISCONN_LEN =18 , -VERR_DISCONN_RQST_LEN =19 , -VERR_DISCONN_CMD =20 , -VERR_DISCONN_CMD_LEN =21 , -VERR_DISCONN_SCOPE =22 , -VERR_RS_LEN =23 , -VERR_RS_RQST_LEN =24 , -VERR_RS_CMD =25 , -VERR_RS_CMD_LEN =26 , -VERR_RS_RCTL =27 , -VERR_RS_RO =28 , -}; - -static char *validation_errors[] = { -"OK" , -"Bad CR_ASSOC Length" , -"Bad CR_ASSOC Rqst Length" , -"Not CR_ASSOC Cmd" , -"Bad CR_ASSOC Cmd Length" , -"Bad Ersp Ratio" , -"Association Allocation Failed" , -"Queue Allocation Failed" , -"Bad CR_CONN Length" , -"Bad CR_CONN Rqst Length" , -"Not Association ID" , -"Bad Association ID Length" , -"No Association" , -"Not Connection ID" , -"Bad Connection ID Length" , -"No Connection" , -"Not CR_CONN Cmd" , -"Bad CR_CONN Cmd Length" , -"Bad DISCONN Length" , -"Bad DISCONN Rqst Length" , -"Not DISCONN Cmd" , -"Bad DISCONN Cmd Length" , -"Bad Disconnect Scope" , -"Bad RS Length" , -"Bad RS Rqst Length" , -"Not RS Cmd" , -"Bad RS Cmd Length" , -"Bad RS R_CTL" , -"Bad RS Relative Offset" , -}; - -static void nvmet_fc_ls_create_association (struct nvmet_fc_tgtport *tgtport,struct nvmet_fc_ls_iod *iod) {@@ -1407,7 +1312,7 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Create Association LS failed: %s\n" , validation_errors[ret]); - iod->lsrsp->rsplen =nvmet_fc_format_rjt (acc, + iod->lsrsp->rsplen =nvme_fc_format_rjt (acc,NVME_FC_MAX_LS_BUFFER_SIZE , rqst->w0.ls_cmd,FCNVME_RJT_RC_LOGIC ,FCNVME_RJT_EXP_NONE ,0 );@@ -1422,7 +1327,7 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport, iod->lsrsp->rsplen =sizeof (*acc); -nvmet_fc_format_rsp_hdr (acc,FCNVME_LS_ACC , +nvme_fc_format_rsp_hdr (acc,FCNVME_LS_ACC ,fcnvme_lsdesc_len (sizeof (struct fcnvme_ls_cr_assoc_acc )),FCNVME_LS_CREATE_ASSOCIATION );@@ -1498,7 +1403,7 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Create Connection LS failed: %s\n" , validation_errors[ret]); - iod->lsrsp->rsplen = nvmet_fc_format_rjt(acc, + iod->lsrsp->rsplen = nvme_fc_format_rjt(acc, NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd, (ret == VERR_NO_ASSOC) ? FCNVME_RJT_RC_INV_ASSOC :@@ -1515,7 +1420,7 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport, iod->lsrsp->rsplen =sizeof (*acc); -nvmet_fc_format_rsp_hdr (acc,FCNVME_LS_ACC , +nvme_fc_format_rsp_hdr (acc,FCNVME_LS_ACC ,fcnvme_lsdesc_len (sizeof (struct fcnvme_ls_cr_conn_acc )),FCNVME_LS_CREATE_CONNECTION ); acc->connectid.desc_tag =cpu_to_be32 (FCNVME_LSDESC_CONN_ID );@@ -1578,13 +1483,11 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,dev_err (tgtport->dev,"Disconnect LS failed: %s\n" , validation_errors[ret]); - iod->lsrsp->rsplen =nvmet_fc_format_rjt (acc, + iod->lsrsp->rsplen =nvme_fc_format_rjt (acc,NVME_FC_MAX_LS_BUFFER_SIZE , rqst->w0.ls_cmd, (ret ==VERR_NO_ASSOC ) ?FCNVME_RJT_RC_INV_ASSOC : - (ret ==VERR_NO_CONN ) ? -FCNVME_RJT_RC_INV_CONN : -FCNVME_RJT_RC_LOGIC , +FCNVME_RJT_RC_LOGIC ,FCNVME_RJT_EXP_NONE ,0 );return ; }@@ -1593,7 +1496,7 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport, iod->lsrsp->rsplen =sizeof (*acc); -nvmet_fc_format_rsp_hdr (acc,FCNVME_LS_ACC , +nvme_fc_format_rsp_hdr (acc,FCNVME_LS_ACC ,fcnvme_lsdesc_len (sizeof (struct fcnvme_ls_disconnect_assoc_acc )),FCNVME_LS_DISCONNECT_ASSOC );
drivers/nvme/target/fc.c nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport, · line 1585
Quoted 8 lines
drivers/nvme/target/fc.c +1
-1@@ -1676,7 +1579,7 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport,nvmet_fc_ls_disconnect (tgtport, iod);break ;default : - iod->lsrsp->rsplen =nvmet_fc_format_rjt (iod->rspbuf, + iod->lsrsp->rsplen =nvme_fc_format_rjt (iod->rspbuf,NVME_FC_MAX_LS_BUFFER_SIZE , w0->ls_cmd,FCNVME_RJT_RC_INVAL ,FCNVME_RJT_EXP_NONE ,0 ); }
Makes sense.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 8/29 nvmet-fc: Better size LS buffers
In reply to James SmartCurrent code uses NVME_FC_MAX_LS_BUFFER_SIZE (2KB) when allocating buffers for LS requests and responses. This is considerable overkill for what is actually defined.
Rework code to have unions for all possible requests and responses and size based on the unions. Remove NVME_FC_MAX_LS_BUFFER_SIZE.
--- drivers/nvme/host/fc.h | 15 ++++++++++++++ drivers/nvme/target/fc.c | 53 +++++++++++++++++++++--------------------------- 2 files changed, 38 insertions(+), 30 deletions(-)
drivers/nvme/host/fc.h +15 -0
@@ -16,6 +16,21 @@* ****************** FC-NVME LS HANDLING ****************** */ +union nvmefc_ls_requests { +struct fcnvme_ls_cr_assoc_rqst rq_cr_assoc; +struct fcnvme_ls_cr_conn_rqst rq_cr_conn; +struct fcnvme_ls_disconnect_assoc_rqst rq_dis_assoc; +struct fcnvme_ls_disconnect_conn_rqst rq_dis_conn; +}__aligned(128) ;/* alignment for other things alloc'd with */ + +union nvmefc_ls_responses { +struct fcnvme_ls_rjt rsp_rjt; +struct fcnvme_ls_cr_assoc_acc rsp_cr_assoc; +struct fcnvme_ls_cr_conn_acc rsp_cr_conn; +struct fcnvme_ls_disconnect_assoc_acc rsp_dis_assoc; +struct fcnvme_ls_disconnect_conn_acc rsp_dis_conn; +}__aligned(128) ;/* alignment for other things alloc'd with */ +static inline void nvme_fc_format_rsp_hdr (void *buf,u8 ls_cmd,__be32 desc_len,u8 rqst_ls_cmd) {
drivers/nvme/target/fc.c +23 -30
155 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 08/29] nvmet-fc: Better size LS buffers
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 08/29] nvmet-fc: Better size LS buffers
In reply to James SmartQuoted 4 lines · James Smart
Current code uses
NVME_FC_MAX_LS_BUFFER_SIZE(2KB) when allocating buffers for LS requests and responses. This is considerable overkill for what is actually defined.
Quoted 9 lines
Rework code to have unions for all possible requests and responses and size based on the unions. Remove
NVME_FC_MAX_LS_BUFFER_SIZE.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.h | 15 ++++++++++++++ drivers/nvme/target/fc.c | 53 +++++++++++++++++++++--------------------------- 2 files changed, 38 insertions(+), 30 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 9/29 nvme-fc: Ensure private pointers are NULL if no data
In reply to James SmartEnsure that when allocations are done, and the lldd options indicate no private data is needed, that private pointers will be set to NULL (catches driver error that forgot to set private data size).
Slightly reorg the allocations so that private data follows allocations for LS request/response buffers. Ensures better alignments for the buffers as well as the private pointer.
--- drivers/nvme/host/fc.c | 81 ++++++++++++++++++++++++++++++------------------ drivers/nvme/target/fc.c | 5 ++- 2 files changed, 54 insertions(+), 32 deletions(-)
drivers/nvme/host/fc.c +50 -31
@@ -396,7 +396,10 @@ nvme_fc_register_localport(struct nvme_fc_port_info *pinfo, newrec->ops = template; newrec->dev = dev;ida_init (&newrec->endp_cnt); - newrec->localport.private = &newrec[1 ]; +if (template->local_priv_sz) + newrec->localport.private = &newrec[1 ]; +else + newrec->localport.private =NULL ; newrec->localport.node_name = pinfo->node_name; newrec->localport.port_name = pinfo->port_name; newrec->localport.port_role = pinfo->port_role;
@@ -705,7 +708,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport, newrec->remoteport.localport = &lport->localport; newrec->dev = lport->dev; newrec->lport = lport; - newrec->remoteport.private = &newrec[1 ]; +if (lport->ops->remote_priv_sz) + newrec->remoteport.private = &newrec[1 ]; +else + newrec->remoteport.private =NULL ; newrec->remoteport.port_role = pinfo->port_role; newrec->remoteport.node_name = pinfo->node_name; newrec->remoteport.port_name = pinfo->port_name;
@@ -1153,18 +1159,23 @@ nvme_fc_connect_admin_queue(struct nvme_fc_ctrl *ctrl,int ret, fcret =0 ; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*assoc_rqst) +sizeof (*assoc_acc)),GFP_KERNEL ); +sizeof (*assoc_rqst) +sizeof (*assoc_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL );if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Create Association failed: ENOMEM\n" , + ctrl->cnum); ret = -ENOMEM ; goto out_no_memory; } - lsreq = &lsop->ls_req; - lsreq->private = (void *)&lsop[1 ]; - assoc_rqst = (struct fcnvme_ls_cr_assoc_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + assoc_rqst = (struct fcnvme_ls_cr_assoc_rqst *)&lsop[1 ]; assoc_acc = (struct fcnvme_ls_cr_assoc_acc *)&assoc_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = &assoc_acc[1 ]; +else + lsreq->private =NULL ; assoc_rqst->w0.ls_cmd =FCNVME_LS_CREATE_ASSOCIATION ; assoc_rqst->desc_list_len =
@@ -1262,18 +1273,23 @@ nvme_fc_connect_queue(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,int ret, fcret =0 ; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*conn_rqst) +sizeof (*conn_acc)),GFP_KERNEL ); +sizeof (*conn_rqst) +sizeof (*conn_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL );if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Create Connection failed: ENOMEM\n" , + ctrl->cnum); ret = -ENOMEM ; goto out_no_memory; } - lsreq = &lsop->ls_req; - lsreq->private = (void *)&lsop[1 ]; - conn_rqst = (struct fcnvme_ls_cr_conn_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + conn_rqst = (struct fcnvme_ls_cr_conn_rqst *)&lsop[1 ]; conn_acc = (struct fcnvme_ls_cr_conn_acc *)&conn_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = (void *)&conn_acc[1 ]; +else + lsreq->private =NULL ; conn_rqst->w0.ls_cmd =FCNVME_LS_CREATE_CONNECTION ; conn_rqst->desc_list_len = cpu_to_be32(
@@ -1387,19 +1403,23 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)int ret; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*discon_rqst) +sizeof (*discon_acc)), -GFP_KERNEL ); -if (!lsop) -/* couldn't sent it... too bad */ +sizeof (*discon_rqst) +sizeof (*discon_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL ); +if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Disconnect Association " +"failed: ENOMEM\n" , + ctrl->cnum);return ; + } - lsreq = &lsop->ls_req; - - lsreq->private = (void *)&lsop[1 ]; - discon_rqst = (struct fcnvme_ls_disconnect_assoc_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + discon_rqst = (struct fcnvme_ls_disconnect_assoc_rqst *)&lsop[1 ]; discon_acc = (struct fcnvme_ls_disconnect_assoc_acc *)&discon_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = (void *)&discon_acc[1 ]; +else + lsreq->private =NULL ; discon_rqst->w0.ls_cmd =FCNVME_LS_DISCONNECT_ASSOC ; discon_rqst->desc_list_len = cpu_to_be32(
@@ -1785,15 +1805,17 @@ nvme_fc_init_aen_ops(struct nvme_fc_ctrl *ctrl)struct nvme_fc_fcp_op *aen_op;struct nvme_fc_cmd_iu *cmdiu;struct nvme_command *sqe; -void *private; +void *private =NULL ;int i, ret; aen_op = ctrl->aen_ops;for (i =0 ; i <NVME_NR_AEN_COMMANDS ; i++, aen_op++) { - private =kzalloc (ctrl->lport->ops->fcprqst_priv_sz, +if (ctrl->lport->ops->fcprqst_priv_sz) { + private =kzalloc (ctrl->lport->ops->fcprqst_priv_sz,GFP_KERNEL ); -if (!private) -return -ENOMEM ; +if (!private) +return -ENOMEM ; + } cmdiu = &aen_op->cmd_iu; sqe = &cmdiu->sqe;
@@ -1824,9 +1846,6 @@ nvme_fc_term_aen_ops(struct nvme_fc_ctrl *ctrl) aen_op = ctrl->aen_ops;for (i =0 ; i <NVME_NR_AEN_COMMANDS ; i++, aen_op++) { -if (!aen_op->fcp_req.private) -continue ; -__nvme_fc_exit_request (ctrl, aen_op);kfree (aen_op->fcp_req.private);
drivers/nvme/target/fc.c +4 -1
@@ -1047,7 +1047,10 @@ nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo, newrec->fc_target_port.node_name = pinfo->node_name; newrec->fc_target_port.port_name = pinfo->port_name; - newrec->fc_target_port.private = &newrec[1 ]; +if (template->target_priv_sz) + newrec->fc_target_port.private = &newrec[1 ]; +else + newrec->fc_target_port.private =NULL ; newrec->fc_target_port.port_id = pinfo->port_id; newrec->fc_target_port.port_num = idx;INIT_LIST_HEAD (&newrec->tgt_list);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 09/29] nvme-fc: Ensure private pointers are NULL if no data
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 09/29] nvme-fc: Ensure private pointers are NULL if no data
In reply to James SmartQuoted 8 lines · James Smart
Ensure that when allocations are done, and the lldd options indicate no private data is needed, that private pointers will be set to NULL (catches driver error that forgot to set private data size).
Slightly reorg the allocations so that private data follows allocations for LS request/response buffers. Ensures better alignments for the buffers as well as the private pointer.
Quoted 6 lines
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 81 ++++++++++++++++++++++++++++++------------------ drivers/nvme/target/fc.c | 5 ++- 2 files changed, 54 insertions(+), 32 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 09/29] nvme-fc: Ensure private pointers are NULL if no data
In reply to James SmartQuoted 175 lines · James Smart
Ensure that when allocations are done, and the lldd options indicate no private data is needed, that private pointers will be set to NULL (catches driver error that forgot to set private data size).
Slightly reorg the allocations so that private data follows allocations for LS request/response buffers. Ensures better alignments for the buffers as well as the private pointer.
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 81 ++++++++++++++++++++++++++++++------------------ drivers/nvme/target/fc.c | 5 ++- 2 files changed, 54 insertions(+), 32 deletions(-)drivers/nvme/host/fc.c +50
-31@@ -396,7 +396,10 @@ nvme_fc_register_localport(struct nvme_fc_port_info *pinfo, newrec->ops = template; newrec->dev = dev;ida_init (&newrec->endp_cnt); - newrec->localport.private = &newrec[1 ]; +if (template->local_priv_sz) + newrec->localport.private = &newrec[1 ]; +else + newrec->localport.private =NULL ; newrec->localport.node_name = pinfo->node_name; newrec->localport.port_name = pinfo->port_name; newrec->localport.port_role = pinfo->port_role;@@ -705,7 +708,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport, newrec->remoteport.localport = &lport->localport; newrec->dev = lport->dev; newrec->lport = lport; - newrec->remoteport.private = &newrec[1 ]; +if (lport->ops->remote_priv_sz) + newrec->remoteport.private = &newrec[1 ]; +else + newrec->remoteport.private =NULL ; newrec->remoteport.port_role = pinfo->port_role; newrec->remoteport.node_name = pinfo->node_name; newrec->remoteport.port_name = pinfo->port_name;@@ -1153,18 +1159,23 @@ nvme_fc_connect_admin_queue(struct nvme_fc_ctrl *ctrl,int ret, fcret =0 ; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*assoc_rqst) +sizeof (*assoc_acc)),GFP_KERNEL ); +sizeof (*assoc_rqst) +sizeof (*assoc_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL );if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Create Association failed: ENOMEM\n" , + ctrl->cnum); ret = -ENOMEM ; goto out_no_memory; } - lsreq = &lsop->ls_req; - lsreq->private = (void *)&lsop[1 ]; - assoc_rqst = (struct fcnvme_ls_cr_assoc_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + assoc_rqst = (struct fcnvme_ls_cr_assoc_rqst *)&lsop[1 ]; assoc_acc = (struct fcnvme_ls_cr_assoc_acc *)&assoc_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = &assoc_acc[1 ]; +else + lsreq->private =NULL ; assoc_rqst->w0.ls_cmd =FCNVME_LS_CREATE_ASSOCIATION ; assoc_rqst->desc_list_len =@@ -1262,18 +1273,23 @@ nvme_fc_connect_queue(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,int ret, fcret =0 ; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*conn_rqst) +sizeof (*conn_acc)),GFP_KERNEL ); +sizeof (*conn_rqst) +sizeof (*conn_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL );if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Create Connection failed: ENOMEM\n" , + ctrl->cnum); ret = -ENOMEM ; goto out_no_memory; } - lsreq = &lsop->ls_req; - lsreq->private = (void *)&lsop[1 ]; - conn_rqst = (struct fcnvme_ls_cr_conn_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + conn_rqst = (struct fcnvme_ls_cr_conn_rqst *)&lsop[1 ]; conn_acc = (struct fcnvme_ls_cr_conn_acc *)&conn_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = (void *)&conn_acc[1 ]; +else + lsreq->private =NULL ; conn_rqst->w0.ls_cmd =FCNVME_LS_CREATE_CONNECTION ; conn_rqst->desc_list_len = cpu_to_be32(@@ -1387,19 +1403,23 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)int ret; lsop =kzalloc ((sizeof (*lsop) + - ctrl->lport->ops->lsrqst_priv_sz + -sizeof (*discon_rqst) +sizeof (*discon_acc)), -GFP_KERNEL ); -if (!lsop) -/* couldn't sent it... too bad */ +sizeof (*discon_rqst) +sizeof (*discon_acc) + + ctrl->lport->ops->lsrqst_priv_sz),GFP_KERNEL ); +if (!lsop) { +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: send Disconnect Association " +"failed: ENOMEM\n" , + ctrl->cnum);return ; + } - lsreq = &lsop->ls_req; - - lsreq->private = (void *)&lsop[1 ]; - discon_rqst = (struct fcnvme_ls_disconnect_assoc_rqst *) - (lsreq->private + ctrl->lport->ops->lsrqst_priv_sz); + discon_rqst = (struct fcnvme_ls_disconnect_assoc_rqst *)&lsop[1 ]; discon_acc = (struct fcnvme_ls_disconnect_assoc_acc *)&discon_rqst[1 ]; + lsreq = &lsop->ls_req; +if (ctrl->lport->ops->lsrqst_priv_sz) + lsreq->private = (void *)&discon_acc[1 ]; +else + lsreq->private =NULL ; discon_rqst->w0.ls_cmd =FCNVME_LS_DISCONNECT_ASSOC ; discon_rqst->desc_list_len = cpu_to_be32(@@ -1785,15 +1805,17 @@ nvme_fc_init_aen_ops(struct nvme_fc_ctrl *ctrl)struct nvme_fc_fcp_op *aen_op;struct nvme_fc_cmd_iu *cmdiu;struct nvme_command *sqe; -void *private; +void *private =NULL ;int i, ret; aen_op = ctrl->aen_ops;for (i =0 ; i <NVME_NR_AEN_COMMANDS ; i++, aen_op++) { - private =kzalloc (ctrl->lport->ops->fcprqst_priv_sz, +if (ctrl->lport->ops->fcprqst_priv_sz) { + private =kzalloc (ctrl->lport->ops->fcprqst_priv_sz,GFP_KERNEL ); -if (!private) -return -ENOMEM ; +if (!private) +return -ENOMEM ; + } cmdiu = &aen_op->cmd_iu; sqe = &cmdiu->sqe;@@ -1824,9 +1846,6 @@ nvme_fc_term_aen_ops(struct nvme_fc_ctrl *ctrl) aen_op = ctrl->aen_ops;for (i =0 ; i <NVME_NR_AEN_COMMANDS ; i++, aen_op++) { -if (!aen_op->fcp_req.private) -continue ; -__nvme_fc_exit_request (ctrl, aen_op);kfree (aen_op->fcp_req.private);drivers/nvme/target/fc.c +0
-0@@ -1047,7 +1047,10 @@ nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo,
drivers/nvme/target/fc.c nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo, · line 1056
Quoted 10 lines
drivers/nvme/target/fc.c +4
-1newrec->fc_target_port.node_name = pinfo->node_name; newrec->fc_target_port.port_name = pinfo->port_name; - newrec->fc_target_port.private = &newrec[1 ]; +if (template->target_priv_sz) + newrec->fc_target_port.private = &newrec[1 ]; +else + newrec->fc_target_port.private =NULL ; newrec->fc_target_port.port_id = pinfo->port_id; newrec->fc_target_port.port_num = idx;INIT_LIST_HEAD (&newrec->tgt_list);
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 10/29 nvmefc: Use common definitions for LS names, formatting, and validation
In reply to James SmartGiven that both host and target now generate and receive LS's create a single table definition for LS names. Each tranport half will have a local version of the table.
As Create Association LS is issued by both sides, and received by both sides, create common routines to format the LS and to validate the LS.
Convert the host side transport to use the new common Create Association LS formatting routine.
Convert the target side transport to use the new common Create Association LS validation routine.
--- drivers/nvme/host/fc.c | 25 ++------------- drivers/nvme/host/fc.h | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 28 ++--------------- 3 files changed, 83 insertions(+), 49 deletions(-)
drivers/nvme/host/fc.c +2 -23
@@ -1421,29 +1421,8 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)else lsreq->private =NULL ; - discon_rqst->w0.ls_cmd =FCNVME_LS_DISCONNECT_ASSOC ; - discon_rqst->desc_list_len =cpu_to_be32 ( -sizeof (struct fcnvme_lsdesc_assoc_id ) + -sizeof (struct fcnvme_lsdesc_disconn_cmd )); - - discon_rqst->associd.desc_tag =cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID ); - discon_rqst->associd.desc_len = -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_assoc_id )); - - discon_rqst->associd.association_id =cpu_to_be64 (ctrl->association_id); - - discon_rqst->discon_cmd.desc_tag =cpu_to_be32 ( -FCNVME_LSDESC_DISCONN_CMD ); - discon_rqst->discon_cmd.desc_len = -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_disconn_cmd )); - - lsreq->rqstaddr = discon_rqst; - lsreq->rqstlen =sizeof (*discon_rqst); - lsreq->rspaddr = discon_acc; - lsreq->rsplen =sizeof (*discon_acc); - lsreq->timeout =NVME_FC_LS_TIMEOUT_SEC ; +nvmefc_fmt_lsreq_discon_assoc (lsreq, discon_rqst, discon_acc, + ctrl->association_id); ret =nvme_fc_send_ls_req_async (ctrl->rport, lsop, nvme_fc_disconnect_assoc_done);
drivers/nvme/host/fc.h +79 -0
89 lines not on this page · raw message
drivers/nvme/target/fc.c +2 -26
@@ -1442,32 +1442,8 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,memset (acc,0 ,sizeof (*acc)); -if (iod->rqstdatalen <sizeof (struct fcnvme_ls_disconnect_assoc_rqst )) - ret =VERR_DISCONN_LEN ; -else if (rqst->desc_list_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_ls_disconnect_assoc_rqst ))) - ret =VERR_DISCONN_RQST_LEN ; -else if (rqst->associd.desc_tag !=cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID )) - ret =VERR_ASSOC_ID ; -else if (rqst->associd.desc_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_assoc_id ))) - ret =VERR_ASSOC_ID_LEN ; -else if (rqst->discon_cmd.desc_tag != -cpu_to_be32 (FCNVME_LSDESC_DISCONN_CMD )) - ret =VERR_DISCONN_CMD ; -else if (rqst->discon_cmd.desc_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_disconn_cmd ))) - ret =VERR_DISCONN_CMD_LEN ; -/* -* As the standard changed on the LS, check if old format and scope -* something other than Association (e.g. 0). -*/ -else if (rqst->discon_cmd.rsvd8[0 ]) - ret =VERR_DISCONN_SCOPE ; -else { + ret =nvmefc_vldt_lsreq_discon_assoc (iod->rqstdatalen, rqst); +if (!ret) {/* match an active association */ assoc =nvmet_fc_find_target_assoc (tgtport,be64_to_cpu (rqst->associd.association_id));
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 10/29] nvmefc: Use common definitions for LS names, formatting, and validation
Quoted 14 lines · James Smart
Given that both host and target now generate and receive LS's create a single table definition for LS names. Each tranport half will have a local version of the table.
As Create Association LS is issued by both sides, and received by both sides, create common routines to format the LS and to validate the LS.
Convert the host side transport to use the new common Create Association LS formatting routine.
Convert the target side transport to use the new common Create Association LS validation routine.
Quoted 7 lines
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 25 ++------------- drivers/nvme/host/fc.h | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 28 ++--------------- 3 files changed, 83 insertions(+), 49 deletions(-)Reviewed-by Hannes Reinecke <[email protected]>
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 10/29] nvmefc: Use common definitions for LS names, formatting, and validation
In reply to James SmartQuoted 182 lines · James Smart
Given that both host and target now generate and receive LS's create a single table definition for LS names. Each tranport half will have a local version of the table.
As Create Association LS is issued by both sides, and received by both sides, create common routines to format the LS and to validate the LS.
Convert the host side transport to use the new common Create Association LS formatting routine.
Convert the target side transport to use the new common Create Association LS validation routine.
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 25 ++------------- drivers/nvme/host/fc.h | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/fc.c | 28 ++--------------- 3 files changed, 83 insertions(+), 49 deletions(-)drivers/nvme/host/fc.c +2
-23@@ -1421,29 +1421,8 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)else lsreq->private =NULL ; - discon_rqst->w0.ls_cmd =FCNVME_LS_DISCONNECT_ASSOC ; - discon_rqst->desc_list_len =cpu_to_be32 ( -sizeof (struct fcnvme_lsdesc_assoc_id ) + -sizeof (struct fcnvme_lsdesc_disconn_cmd )); - - discon_rqst->associd.desc_tag =cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID ); - discon_rqst->associd.desc_len = -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_assoc_id )); - - discon_rqst->associd.association_id =cpu_to_be64 (ctrl->association_id); - - discon_rqst->discon_cmd.desc_tag =cpu_to_be32 ( -FCNVME_LSDESC_DISCONN_CMD ); - discon_rqst->discon_cmd.desc_len = -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_disconn_cmd )); - - lsreq->rqstaddr = discon_rqst; - lsreq->rqstlen =sizeof (*discon_rqst); - lsreq->rspaddr = discon_acc; - lsreq->rsplen =sizeof (*discon_acc); - lsreq->timeout =NVME_FC_LS_TIMEOUT_SEC ; +nvmefc_fmt_lsreq_discon_assoc (lsreq, discon_rqst, discon_acc, + ctrl->association_id); ret =nvme_fc_send_ls_req_async (ctrl->rport, lsop, nvme_fc_disconnect_assoc_done);drivers/nvme/host/fc.h +79
-0@@ -17,6 +17,7 @@*/ union nvmefc_ls_requests { +struct fcnvme_ls_rqst_w0 w0;struct fcnvme_ls_cr_assoc_rqst rq_cr_assoc;struct fcnvme_ls_cr_conn_rqst rq_cr_conn;struct fcnvme_ls_disconnect_assoc_rqst rq_dis_assoc;@@ -145,4 +146,82 @@ static char *validation_errors[] = { "Bad Disconnect ACC Length", }; +#define NVME_FC_LAST_LS_CMD_VALUE FCNVME_LS_DISCONNECT_CONN + +static char *nvmefc_ls_names[] = { +"Reserved (0)" , +"RJT (1)" , +"ACC (2)" , +"Create Association" , +"Create Connection" , +"Disconnect Association" , +"Disconnect Connection" , +}; + +static inline void +nvmefc_fmt_lsreq_discon_assoc (struct nvmefc_ls_req *lsreq, +struct fcnvme_ls_disconnect_assoc_rqst *discon_rqst, +struct fcnvme_ls_disconnect_assoc_acc *discon_acc, +u64 association_id) +{ + lsreq->rqstaddr = discon_rqst; + lsreq->rqstlen =sizeof (*discon_rqst); + lsreq->rspaddr = discon_acc; + lsreq->rsplen =sizeof (*discon_acc); + lsreq->timeout =NVME_FC_LS_TIMEOUT_SEC ; + + discon_rqst->w0.ls_cmd =FCNVME_LS_DISCONNECT_ASSOC ; + discon_rqst->desc_list_len =cpu_to_be32 ( +sizeof (struct fcnvme_lsdesc_assoc_id ) + +sizeof (struct fcnvme_lsdesc_disconn_cmd )); + + discon_rqst->associd.desc_tag =cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID ); + discon_rqst->associd.desc_len = +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_lsdesc_assoc_id )); + + discon_rqst->associd.association_id =cpu_to_be64 (association_id); + + discon_rqst->discon_cmd.desc_tag =cpu_to_be32 ( +FCNVME_LSDESC_DISCONN_CMD ); + discon_rqst->discon_cmd.desc_len = +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_lsdesc_disconn_cmd )); +} + +static inline int +nvmefc_vldt_lsreq_discon_assoc (u32 rqstlen, +struct fcnvme_ls_disconnect_assoc_rqst *rqst) +{ +int ret =0 ; + +if (rqstlen <sizeof (struct fcnvme_ls_disconnect_assoc_rqst )) + ret =VERR_DISCONN_LEN ; +else if (rqst->desc_list_len != +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_ls_disconnect_assoc_rqst ))) + ret =VERR_DISCONN_RQST_LEN ; +else if (rqst->associd.desc_tag !=cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID )) + ret =VERR_ASSOC_ID ; +else if (rqst->associd.desc_len != +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_lsdesc_assoc_id ))) + ret =VERR_ASSOC_ID_LEN ; +else if (rqst->discon_cmd.desc_tag != +cpu_to_be32 (FCNVME_LSDESC_DISCONN_CMD )) + ret =VERR_DISCONN_CMD ; +else if (rqst->discon_cmd.desc_len != +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_lsdesc_disconn_cmd ))) + ret =VERR_DISCONN_CMD_LEN ; +/* +* As the standard changed on the LS, check if old format and scope +* something other than Association (e.g. 0). +*/ +else if (rqst->discon_cmd.rsvd8[0 ]) + ret =VERR_DISCONN_SCOPE ; + +return ret; +} +#endif /* _NVME_FC_TRANSPORT_H */ drivers/nvme/target/fc.c +0
-21@@ -1442,32 +1442,8 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,memset (acc,0 ,sizeof (*acc)); -if (iod->rqstdatalen <sizeof (struct fcnvme_ls_disconnect_assoc_rqst )) - ret =VERR_DISCONN_LEN ; -else if (rqst->desc_list_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_ls_disconnect_assoc_rqst ))) - ret =VERR_DISCONN_RQST_LEN ; -else if (rqst->associd.desc_tag !=cpu_to_be32 (FCNVME_LSDESC_ASSOC_ID )) - ret =VERR_ASSOC_ID ; -else if (rqst->associd.desc_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_assoc_id ))) - ret =VERR_ASSOC_ID_LEN ; -else if (rqst->discon_cmd.desc_tag != -cpu_to_be32 (FCNVME_LSDESC_DISCONN_CMD )) - ret =VERR_DISCONN_CMD ; -else if (rqst->discon_cmd.desc_len != -fcnvme_lsdesc_len ( -sizeof (struct fcnvme_lsdesc_disconn_cmd ))) - ret =VERR_DISCONN_CMD_LEN ; -/* -* As the standard changed on the LS, check if old format and scope
drivers/nvme/target/fc.c nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport, · line 1449
Quoted 10 lines
drivers/nvme/target/fc.c +2
-5-* something other than Association (e.g. 0). -*/ - else if (rqst->discon_cmd.rsvd8[0]) - ret =VERR_DISCONN_SCOPE ; - else { + ret =nvmefc_vldt_lsreq_discon_assoc (iod->rqstdatalen, rqst); +if (!ret) {/* match an active association */ assoc =nvmet_fc_find_target_assoc (tgtport,be64_to_cpu (rqst->associd.association_id));
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 11/29 nvme-fc: convert assoc_active flag to atomic
In reply to James SmartConvert the assoc_active flag to an atomic to remove any small race conditions on transitioning to active and back.
--- drivers/nvme/host/fc.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
drivers/nvme/host/fc.c +16 -7
@@ -131,6 +131,11 @@ enum nvme_fcctrl_flags { FCCTRL_TERMIO = (1 << 0), }; +enum { +ASSOC_INACTIVE =0 , +ASSOC_ACTIVE =1 , +}; +struct nvme_fc_ctrl {spinlock_t lock;struct nvme_fc_queue *queues;
@@ -140,7 +145,7 @@ struct nvme_fc_ctrl {u32 cnum;bool ioq_live; -bool assoc_active; +atomic_t assoc_active;atomic_t err_work_active;u64 association_id;
@@ -2584,12 +2589,14 @@ static int nvme_fc_ctlr_active_on_rport(struct nvme_fc_ctrl *ctrl) {struct nvme_fc_rport *rport = ctrl->rport; +int priorstate;u32 cnt; -if (ctrl->assoc_active) + priorstate =atomic_cmpxchg (&ctrl->assoc_active, +ASSOC_INACTIVE ,ASSOC_ACTIVE ); +if (priorstate !=ASSOC_INACTIVE )return 1 ; - ctrl->assoc_active = true; cnt =atomic_inc_return (&rport->act_ctrl_cnt);if (cnt ==1 )nvme_fc_rport_active_on_lport (rport);
@@ -2746,7 +2753,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)__nvme_fc_delete_hw_queue (ctrl, &ctrl->queues[0 ],0 ); out_free_queue:nvme_fc_free_queue (&ctrl->queues[0 ]); - ctrl->assoc_active = false; +atomic_set (&ctrl->assoc_active,ASSOC_INACTIVE );nvme_fc_ctlr_inactive_on_rport (ctrl);return ret;
@@ -2762,10 +2769,12 @@ static void nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl) {unsigned long flags; +int priorstate; -if (!ctrl->assoc_active) + priorstate =atomic_cmpxchg (&ctrl->assoc_active, +ASSOC_ACTIVE ,ASSOC_INACTIVE ); +if (priorstate !=ASSOC_ACTIVE )return ; - ctrl->assoc_active = false;spin_lock_irqsave (&ctrl->lock, flags); ctrl->flags |=FCCTRL_TERMIO ;
@@ -3096,7 +3105,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, ctrl->dev = lport->dev; ctrl->cnum = idx; ctrl->ioq_live = false; - ctrl->assoc_active = false; +atomic_set (&ctrl->assoc_active,ASSOC_INACTIVE );atomic_set (&ctrl->err_work_active,0 );init_waitqueue_head (&ctrl->ioabort_wait);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 11/29] nvme-fc: convert assoc_active flag to atomic
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 11/29] nvme-fc: convert assoc_active flag to atomic
In reply to James SmartQuoted 8 lines · James Smart
Convert the
assoc_activeflag to an atomic to remove any small race conditions on transitioning to active and back.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
As it's just a single value, wouldn't 'test_and_set_bit' work as well here? It could even merged with 'ioq_live' and 'err_work_active' to save some space here ...
Might even be beneficial for performance; atomics always imply a full barrier, and I tend to think that 'test_and_set_bit' is slightly better in that regard.
I might be mistaken, though :-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 11/29] nvme-fc: convert assoc_active flag to atomic
In reply to James SmartQuoted 76 lines · James Smart
Convert the
assoc_activeflag to an atomic to remove any small race conditions on transitioning to active and back.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)drivers/nvme/host/fc.c +15
-6@@ -131,6 +131,11 @@ enum nvme_fcctrl_flags { FCCTRL_TERMIO = (1 << 0), }; +enum { +ASSOC_INACTIVE =0 , +ASSOC_ACTIVE =1 , +}; +struct nvme_fc_ctrl {spinlock_t lock;struct nvme_fc_queue *queues;@@ -140,7 +145,7 @@ struct nvme_fc_ctrl {u32 cnum;bool ioq_live; -bool assoc_active; +atomic_t assoc_active;atomic_t err_work_active;u64 association_id;@@ -2584,12 +2589,14 @@ static int nvme_fc_ctlr_active_on_rport(struct nvme_fc_ctrl *ctrl) {struct nvme_fc_rport *rport = ctrl->rport; +int priorstate;u32 cnt; -if (ctrl->assoc_active) + priorstate =atomic_cmpxchg (&ctrl->assoc_active, +ASSOC_INACTIVE ,ASSOC_ACTIVE ); +if (priorstate !=ASSOC_INACTIVE )return 1 ; - ctrl->assoc_active = true; cnt =atomic_inc_return (&rport->act_ctrl_cnt);if (cnt ==1 )nvme_fc_rport_active_on_lport (rport);@@ -2746,7 +2753,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)__nvme_fc_delete_hw_queue (ctrl, &ctrl->queues[0 ],0 ); out_free_queue:nvme_fc_free_queue (&ctrl->queues[0 ]); - ctrl->assoc_active = false; +atomic_set (&ctrl->assoc_active,ASSOC_INACTIVE );nvme_fc_ctlr_inactive_on_rport (ctrl);return ret;@@ -2762,10 +2769,12 @@ static void nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl) {unsigned long flags; +int priorstate; -if (!ctrl->assoc_active) + priorstate =atomic_cmpxchg (&ctrl->assoc_active, +ASSOC_ACTIVE ,ASSOC_INACTIVE ); +if (priorstate !=ASSOC_ACTIVE )return ; - ctrl->assoc_active = false;spin_lock_irqsave (&ctrl->lock, flags); ctrl->flags |=FCCTRL_TERMIO ;
drivers/nvme/host/fc.c nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, · line 3111
Quoted 8 lines
drivers/nvme/host/fc.c +1
-1@@ -3096,7 +3105,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, ctrl->dev = lport->dev; ctrl->cnum = idx; ctrl->ioq_live = false; - ctrl->assoc_active = false; +atomic_set (&ctrl->assoc_active,ASSOC_INACTIVE );atomic_set (&ctrl->err_work_active,0 );init_waitqueue_head (&ctrl->ioabort_wait);
Looks Good
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 12/29 nvme-fc: Add Disconnect Association Rcv support
In reply to James SmartThe nvme-fc host transport did not support the reception of a FC-NVME LS. Reception is necessary to implement full compliance with FC-NVME-2.
Populate the LS receive handler, and specifically the handling of a Disconnect Association LS. The response to the LS, if it matched a controller, must be sent after the aborts for any I/O on any connection have been sent.
--- drivers/nvme/host/fc.c | 363 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 359 insertions(+), 4 deletions(-)
drivers/nvme/host/fc.c +359 -4
@@ -62,6 +62,17 @@ struct nvmefc_ls_req_op {bool req_queued; }; +struct nvmefc_ls_rcv_op { +struct nvme_fc_rport *rport; +struct nvmefc_ls_rsp *lsrsp; +union nvmefc_ls_requests *rqstbuf; +union nvmefc_ls_responses *rspbuf; +u16 rqstdatalen; +bool handled; +dma_addr_t rspdma; +struct list_head lsrcv_list;/* rport->ls_rcv_list */ +}__aligned(sizeof(u64)) ;/* alignment for other things alloc'd with */ +enum nvme_fcpop_state {FCPOP_STATE_UNINIT =0 ,FCPOP_STATE_IDLE =1 ,
@@ -118,6 +129,7 @@ struct nvme_fc_rport {struct list_head endp_list;/* for lport->endp_list */ struct list_head ctrl_list;struct list_head ls_req_list; +struct list_head ls_rcv_list;struct list_head disc_list;struct device *dev;/* physical device for dma */ struct nvme_fc_lport *lport;
@@ -125,6 +137,7 @@ struct nvme_fc_rport {struct kref ref;atomic_t act_ctrl_cnt;unsigned long dev_loss_end; +struct work_struct lsrcv_work; }__aligned(sizeof(u64)) ;/* alignment for other things alloc'd with */ enum nvme_fcctrl_flags {
@@ -148,6 +161,7 @@ struct nvme_fc_ctrl {atomic_t assoc_active;atomic_t err_work_active;u64 association_id; +struct nvmefc_ls_rcv_op *rcv_disconn;struct list_head ctrl_list;/* rport->ctrl_list */
@@ -225,6 +239,9 @@ static struct device *fc_udev_device;static void __nvme_fc_delete_hw_queue (struct nvme_fc_ctrl *,struct nvme_fc_queue *,unsigned int ); +static void nvme_fc_handle_ls_rqst_work (struct work_struct *work); + +static void nvme_fc_free_lport (struct kref *ref) {
@@ -711,6 +728,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,atomic_set (&newrec->act_ctrl_cnt,0 );spin_lock_init (&newrec->lock); newrec->remoteport.localport = &lport->localport; +INIT_LIST_HEAD (&newrec->ls_rcv_list); newrec->dev = lport->dev; newrec->lport = lport;if (lport->ops->remote_priv_sz)
@@ -724,6 +742,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport, newrec->remoteport.port_state =FC_OBJSTATE_ONLINE ; newrec->remoteport.port_num = idx;__nvme_fc_set_dev_loss_tmo (newrec, pinfo); +INIT_WORK (&newrec->lsrcv_work, nvme_fc_handle_ls_rqst_work);spin_lock_irqsave (&nvme_fc_lock, flags);list_add_tail (&newrec->endp_list, &lport->endp_list);
@@ -1013,6 +1032,7 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,static void nvme_fc_ctrl_put (struct nvme_fc_ctrl *);static int nvme_fc_ctrl_get (struct nvme_fc_ctrl *); +static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg);static void __nvme_fc_finish_ls_req (struct nvmefc_ls_req_op *lsop)
@@ -1161,6 +1181,7 @@ nvme_fc_connect_admin_queue(struct nvme_fc_ctrl *ctrl,struct nvmefc_ls_req *lsreq;struct fcnvme_ls_cr_assoc_rqst *assoc_rqst;struct fcnvme_ls_cr_assoc_acc *assoc_acc; +unsigned long flags;int ret, fcret =0 ; lsop = kzalloc((sizeof(*lsop) +
@@ -1250,11 +1271,13 @@ nvme_fc_connect_admin_queue(struct nvme_fc_ctrl *ctrl,"q %d Create Association LS failed: %s\n" , queue->qnum, validation_errors[fcret]); }else { +spin_lock_irqsave (&ctrl->lock, flags); ctrl->association_id =be64_to_cpu (assoc_acc->associd.association_id); queue->connection_id =be64_to_cpu (assoc_acc->connectid.connection_id);set_bit (NVME_FC_Q_CONNECTED , &queue->flags); +spin_unlock_irqrestore (&ctrl->lock, flags); } out_free_buffer:
@@ -1435,6 +1458,247 @@ nvme_fc_xmt_disconnect_assoc(struct nvme_fc_ctrl *ctrl)kfree (lsop); } +static void +nvme_fc_xmt_ls_rsp_done (struct nvmefc_ls_rsp *lsrsp) +{ +struct nvmefc_ls_rcv_op *lsop = lsrsp->nvme_fc_private; +struct nvme_fc_rport *rport = lsop->rport; +struct nvme_fc_lport *lport = rport->lport; +unsigned long flags; + +spin_lock_irqsave (&rport->lock, flags); +list_del (&lsop->lsrcv_list); +spin_unlock_irqrestore (&rport->lock, flags); + +fc_dma_sync_single_for_cpu (lport->dev, lsop->rspdma, +sizeof (*lsop->rspbuf),DMA_TO_DEVICE ); +fc_dma_unmap_single (lport->dev, lsop->rspdma, +sizeof (*lsop->rspbuf),DMA_TO_DEVICE ); + +kfree (lsop); + +nvme_fc_rport_put (rport); +} + +static void +nvme_fc_xmt_ls_rsp (struct nvmefc_ls_rcv_op *lsop) +{ +struct nvme_fc_rport *rport = lsop->rport; +struct nvme_fc_lport *lport = rport->lport; +struct fcnvme_ls_rqst_w0 *w0 = &lsop->rqstbuf->w0; +int ret; + +fc_dma_sync_single_for_device (lport->dev, lsop->rspdma, +sizeof (*lsop->rspbuf),DMA_TO_DEVICE ); + + ret = lport->ops->xmt_ls_rsp (&lport->localport, &rport->remoteport, + lsop->lsrsp); +if (ret) { +dev_warn (lport->dev, +"LLDD rejected LS RSP xmt: LS %d status %d\n" , + w0->ls_cmd, ret); +nvme_fc_xmt_ls_rsp_done (lsop->lsrsp); +return ; + } +} + +static struct nvme_fc_ctrl * +nvme_fc_match_disconn_ls (struct nvme_fc_rport *rport, +struct nvmefc_ls_rcv_op *lsop) +{ +struct fcnvme_ls_disconnect_assoc_rqst *rqst = + &lsop->rqstbuf->rq_dis_assoc; +struct nvme_fc_ctrl *ctrl, *ret =NULL ; +struct nvmefc_ls_rcv_op *oldls =NULL ; +u64 association_id =be64_to_cpu (rqst->associd.association_id); +unsigned long flags; + +spin_lock_irqsave (&rport->lock, flags); + +list_for_each_entry (ctrl, &rport->ctrl_list, ctrl_list) { +if (!nvme_fc_ctrl_get (ctrl)) +continue ; +spin_lock (&ctrl->lock); +if (association_id == ctrl->association_id) { + oldls = ctrl->rcv_disconn; + ctrl->rcv_disconn = lsop; + ret = ctrl; + } +spin_unlock (&ctrl->lock); +if (ret) +/* leave the ctrl get reference */ +break ; +nvme_fc_ctrl_put (ctrl); + } + +spin_unlock_irqrestore (&rport->lock, flags); + +/* transmit a response for anything that was pending */ +if (oldls) { +dev_info (rport->lport->dev, +"NVME-FC{%d}: Multiple Disconnect Association " +"LS's received\n" , ctrl->cnum); +/* overwrite good response with bogus failure */ + oldls->lsrsp->rsplen =nvme_fc_format_rjt (oldls->rspbuf, +sizeof (*oldls->rspbuf), + rqst->w0.ls_cmd, +FCNVME_RJT_RC_UNAB , +FCNVME_RJT_EXP_NONE ,0 ); +nvme_fc_xmt_ls_rsp (oldls); + } + +return ret; +} + +/* +* returns true to mean LS handled and ls_rsp can be sent +* returns false to defer ls_rsp xmt (will be done as part of +* association termination) +*/ +static bool +nvme_fc_ls_disconnect_assoc (struct nvmefc_ls_rcv_op *lsop) +{ +struct nvme_fc_rport *rport = lsop->rport; +struct fcnvme_ls_disconnect_assoc_rqst *rqst = + &lsop->rqstbuf->rq_dis_assoc; +struct fcnvme_ls_disconnect_assoc_acc *acc = + &lsop->rspbuf->rsp_dis_assoc; +struct nvme_fc_ctrl *ctrl =NULL ; +int ret =0 ; + +memset (acc,0 ,sizeof (*acc)); + + ret =nvmefc_vldt_lsreq_discon_assoc (lsop->rqstdatalen, rqst); +if (!ret) { +/* match an active association */ + ctrl =nvme_fc_match_disconn_ls (rport, lsop); +if (!ctrl) + ret =VERR_NO_ASSOC ; + } + +if (ret) { +dev_info (rport->lport->dev, +"Disconnect LS failed: %s\n" , + validation_errors[ret]); + lsop->lsrsp->rsplen =nvme_fc_format_rjt (acc, +sizeof (*acc), rqst->w0.ls_cmd, + (ret ==VERR_NO_ASSOC ) ? +FCNVME_RJT_RC_INV_ASSOC : +FCNVME_RJT_RC_LOGIC , +FCNVME_RJT_EXP_NONE ,0 ); +return true; + } + +/* format an ACCept response */ + + lsop->lsrsp->rsplen =sizeof (*acc); + +nvme_fc_format_rsp_hdr (acc,FCNVME_LS_ACC , +fcnvme_lsdesc_len ( +sizeof (struct fcnvme_ls_disconnect_assoc_acc )), +FCNVME_LS_DISCONNECT_ASSOC ); + +/* +* the transmit of the response will occur after the exchanges +* for the association have been ABTS'd by +* nvme_fc_delete_association(). +*/ + +/* fail the association */ +nvme_fc_error_recovery (ctrl,"Disconnect Association LS received" ); + +/* release the reference taken by nvme_fc_match_disconn_ls() */ +nvme_fc_ctrl_put (ctrl); + +return false; +} + +/* +* Actual Processing routine for received FC-NVME LS Requests from the LLD +* returns true if a response should be sent afterward, false if rsp will +* be sent asynchronously. +*/ +static bool +nvme_fc_handle_ls_rqst (struct nvmefc_ls_rcv_op *lsop) +{ +struct fcnvme_ls_rqst_w0 *w0 = &lsop->rqstbuf->w0; +bool ret = true; + + lsop->lsrsp->nvme_fc_private = lsop; + lsop->lsrsp->rspbuf = lsop->rspbuf; + lsop->lsrsp->rspdma = lsop->rspdma; + lsop->lsrsp->done = nvme_fc_xmt_ls_rsp_done; +/* Be preventative. handlers will later set to valid length */ + lsop->lsrsp->rsplen =0 ; + +/* +* handlers: +* parse request input, execute the request, and format the +* LS response +*/ +switch (w0->ls_cmd) { +case FCNVME_LS_DISCONNECT_ASSOC : + ret =nvme_fc_ls_disconnect_assoc (lsop); +break ; +case FCNVME_LS_DISCONNECT_CONN : + lsop->lsrsp->rsplen =nvme_fc_format_rjt (lsop->rspbuf, +sizeof (*lsop->rspbuf), w0->ls_cmd, +FCNVME_RJT_RC_UNSUP ,FCNVME_RJT_EXP_NONE ,0 ); +break ; +case FCNVME_LS_CREATE_ASSOCIATION : +case FCNVME_LS_CREATE_CONNECTION : + lsop->lsrsp->rsplen =nvme_fc_format_rjt (lsop->rspbuf, +sizeof (*lsop->rspbuf), w0->ls_cmd, +FCNVME_RJT_RC_LOGIC ,FCNVME_RJT_EXP_NONE ,0 ); +break ; +default : + lsop->lsrsp->rsplen =nvme_fc_format_rjt (lsop->rspbuf, +sizeof (*lsop->rspbuf), w0->ls_cmd, +FCNVME_RJT_RC_INVAL ,FCNVME_RJT_EXP_NONE ,0 ); +break ; + } + +return (ret); +} + +static void +nvme_fc_handle_ls_rqst_work (struct work_struct *work) +{ +struct nvme_fc_rport *rport = + container_of(work, struct nvme_fc_rport, lsrcv_work); +struct fcnvme_ls_rqst_w0 *w0; +struct nvmefc_ls_rcv_op *lsop; +unsigned long flags; +bool sendrsp; + +restart: + sendrsp = true; +spin_lock_irqsave (&rport->lock, flags); +list_for_each_entry (lsop, &rport->ls_rcv_list, lsrcv_list) { +if (lsop->handled) +continue ; + + lsop->handled = true; +if (rport->remoteport.port_state ==FC_OBJSTATE_ONLINE ) { +spin_unlock_irqrestore (&rport->lock, flags); + sendrsp =nvme_fc_handle_ls_rqst (lsop); + }else { +spin_unlock_irqrestore (&rport->lock, flags); + w0 = &lsop->rqstbuf->w0; + lsop->lsrsp->rsplen =nvme_fc_format_rjt ( + lsop->rspbuf, +sizeof (*lsop->rspbuf), + w0->ls_cmd, +FCNVME_RJT_RC_UNAB , +FCNVME_RJT_EXP_NONE ,0 ); + } +if (sendrsp) +nvme_fc_xmt_ls_rsp (lsop); + goto restart; + } +spin_unlock_irqrestore (&rport->lock, flags); +} +/** * nvme_fc_rcv_ls_req - transport entry point called by an LLDD * upon the reception of a NVME LS request.
@@ -1461,20 +1725,92 @@ nvme_fc_rcv_ls_req(struct nvme_fc_remote_port *portptr, {struct nvme_fc_rport *rport =remoteport_to_rport (portptr);struct nvme_fc_lport *lport = rport->lport; +struct fcnvme_ls_rqst_w0 *w0 = (struct fcnvme_ls_rqst_w0 *)lsreqbuf; +struct nvmefc_ls_rcv_op *lsop; +unsigned long flags; +int ret; + +nvme_fc_rport_get (rport);/* validate there's a routine to transmit a response */ -if (!lport->ops->xmt_ls_rsp) -return (-EINVAL ); +if (!lport->ops->xmt_ls_rsp) { +dev_info (lport->dev, +"RCV %s LS failed: no LLDD xmt_ls_rsp\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" ); + ret = -EINVAL ; + goto out_put; + } + +if (lsreqbuf_len >sizeof (union nvmefc_ls_requests )) { +dev_info (lport->dev, +"RCV %s LS failed: payload too large\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" ); + ret = -E2BIG ; + goto out_put; + } + + lsop =kzalloc (sizeof (*lsop) + +sizeof (union nvmefc_ls_requests ) + +sizeof (union nvmefc_ls_responses ), +GFP_KERNEL ); +if (!lsop) { +dev_info (lport->dev, +"RCV %s LS failed: No memory\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" ); + ret = -ENOMEM ; + goto out_put; + } + lsop->rqstbuf = (union nvmefc_ls_requests *)&lsop[1 ]; + lsop->rspbuf = (union nvmefc_ls_responses *)&lsop->rqstbuf[1 ]; + + lsop->rspdma =fc_dma_map_single (lport->dev, lsop->rspbuf, +sizeof (*lsop->rspbuf), +DMA_TO_DEVICE ); +if (fc_dma_mapping_error (lport->dev, lsop->rspdma)) { +dev_info (lport->dev, +"RCV %s LS failed: DMA mapping failure\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" ); + ret = -EFAULT ; + goto out_free; + } + + lsop->rport = rport; + lsop->lsrsp = lsrsp; + +memcpy (lsop->rqstbuf, lsreqbuf, lsreqbuf_len); + lsop->rqstdatalen = lsreqbuf_len; + +spin_lock_irqsave (&rport->lock, flags); +if (rport->remoteport.port_state !=FC_OBJSTATE_ONLINE ) { +spin_unlock_irqrestore (&rport->lock, flags); + ret = -ENOTCONN ; + goto out_unmap; + } +list_add_tail (&lsop->lsrcv_list, &rport->ls_rcv_list); +spin_unlock_irqrestore (&rport->lock, flags); + +schedule_work (&rport->lsrcv_work);return 0 ; + +out_unmap: +fc_dma_unmap_single (lport->dev, lsop->rspdma, +sizeof (*lsop->rspbuf),DMA_TO_DEVICE ); +out_free: +kfree (lsop); +out_put: +nvme_fc_rport_put (rport); +return ret; }EXPORT_SYMBOL_GPL (nvme_fc_rcv_ls_req);/* *********************** NVME Ctrl Routines **************************** */ -static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg); -static void __nvme_fc_exit_request (struct nvme_fc_ctrl *ctrl,struct nvme_fc_fcp_op *op)
@@ -2631,6 +2967,8 @@ static int nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) {struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; +struct nvmefc_ls_rcv_op *disls =NULL ; +unsigned long flags;int ret;bool changed;
@@ -2748,7 +3086,13 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) out_disconnect_admin_queue:/* send a Disconnect(association) LS to fc-nvme target */ nvme_fc_xmt_disconnect_assoc (ctrl); +spin_lock_irqsave (&ctrl->lock, flags); ctrl->association_id =0 ; + disls = ctrl->rcv_disconn; + ctrl->rcv_disconn =NULL ; +spin_unlock_irqrestore (&ctrl->lock, flags); +if (disls) +nvme_fc_xmt_ls_rsp (disls); out_delete_hw_queue:__nvme_fc_delete_hw_queue (ctrl, &ctrl->queues[0 ],0 ); out_free_queue:
@@ -2768,6 +3112,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)static void nvme_fc_delete_association (struct nvme_fc_ctrl *ctrl) { +struct nvmefc_ls_rcv_op *disls =NULL ;unsigned long flags;int priorstate;
@@ -2842,7 +3187,17 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)if (ctrl->association_id)nvme_fc_xmt_disconnect_assoc (ctrl); +spin_lock_irqsave (&ctrl->lock, flags); ctrl->association_id =0 ; + disls = ctrl->rcv_disconn; + ctrl->rcv_disconn =NULL ; +spin_unlock_irqrestore (&ctrl->lock, flags); +if (disls) +/* +* if a Disconnect Request was waiting for a response, send +* now that all ABTS's have been issued (and are complete). +*/ +nvme_fc_xmt_ls_rsp (disls);if (ctrl->ctrl.tagset) {nvme_fc_delete_hw_io_queues (ctrl);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 12/29] nvme-fc: Add Disconnect Association Rcv support
Quoted 9 lines · James Smart
The nvme-fc host transport did not support the reception of a FC-NVME LS. Reception is necessary to implement full compliance with FC-NVME-2.
Populate the LS receive handler, and specifically the handling of a Disconnect Association LS. The response to the LS, if it matched a controller, must be sent after the aborts for any I/O on any connection have been sent.
Quoted 5 lines
Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 363 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 359 insertions(+), 4 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 13/29 nvmet-fc: add LS failure messages
In reply to James SmartAdd LS reception failure messages
--- drivers/nvme/target/fc.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
drivers/nvme/target/fc.c +19 -3
@@ -1598,15 +1598,31 @@ nvmet_fc_rcv_ls_req(struct nvmet_fc_target_port *target_port, {struct nvmet_fc_tgtport *tgtport =targetport_to_tgtport (target_port);struct nvmet_fc_ls_iod *iod; - -if (lsreqbuf_len >sizeof (union nvmefc_ls_requests )) +struct fcnvme_ls_rqst_w0 *w0 = (struct fcnvme_ls_rqst_w0 *)lsreqbuf; + +if (lsreqbuf_len >sizeof (union nvmefc_ls_requests )) { +dev_info (tgtport->dev, +"RCV %s LS failed: payload too large (%d)\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" , + lsreqbuf_len);return -E2BIG ; + } -if (!nvmet_fc_tgtport_get (tgtport)) +if (!nvmet_fc_tgtport_get (tgtport)) { +dev_info (tgtport->dev, +"RCV %s LS failed: target deleting\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" );return -ESHUTDOWN ; + } iod =nvmet_fc_alloc_ls_iod (tgtport);if (!iod) { +dev_info (tgtport->dev, +"RCV %s LS failed: context allocation failed\n" , + (w0->ls_cmd <=NVME_FC_LAST_LS_CMD_VALUE ) ? + nvmefc_ls_names[w0->ls_cmd] :"" );nvmet_fc_tgtport_put (tgtport);return -ENOENT ; }
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 13/29] nvmet-fc: add LS failure messages
Quoted 7 lines · James Smart
Add LS reception failure messages
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 14/29 nvmet-fc: perform small cleanups on unneeded checks
In reply to James SmartWhile code reviewing saw a couple of items that can be cleaned up:
- In nvmet_fc_delete_target_queue(), the routine unlocks, then checks and relocks. Reorganize to avoid the unlock/relock.
- In nvmet_fc_delete_target_queue(), there's a check on the disconnect state that is unnecessary as the routine validates the state before starting any action.
--- drivers/nvme/target/fc.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
drivers/nvme/target/fc.c +4 -7
@@ -688,20 +688,18 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue)if (fod->active) {spin_lock (&fod->flock); fod->abort = true; - writedataactive = fod->writedataactive; -spin_unlock (&fod->flock);/* * only call lldd abort routine if waiting for * writedata. other outstanding ops should finish * on their own. */ -if (writedataactive) { -spin_lock (&fod->flock); +if (fod->writedataactive) { fod->aborted = true;spin_unlock (&fod->flock); tgtport->ops->fcp_abort ( &tgtport->fc_target_port, fod->fcpreq); - } + }else +spin_unlock (&fod->flock); } }
@@ -741,8 +739,7 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue)flush_workqueue (queue->work_q); -if (disconnect) -nvmet_sq_destroy (&queue->nvme_sq); +nvmet_sq_destroy (&queue->nvme_sq);nvmet_fc_tgt_q_put (queue); }
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 14/29] nvmet-fc: perform small cleanups on unneeded checks
Quoted 30 lines · James Smart
While code reviewing saw a couple of items that can be cleaned up:
- Innvmet_fc_delete_target_queue(), the routine unlocks, then checks and relocks. Reorganize to avoid the unlock/relock.
- Innvmet_fc_delete_target_queue(), there's a check on the disconnect state that is unnecessary as the routine validates the state before starting any action.Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)drivers/nvme/target/fc.c +1
-4@@ -688,20 +688,18 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue)if (fod->active) {spin_lock (&fod->flock); fod->abort = true; - writedataactive = fod->writedataactive; -spin_unlock (&fod->flock);/* * only call lldd abort routine if waiting for * writedata. other outstanding ops should finish * on their own. */ -if (writedataactive) { -spin_lock (&fod->flock); +if (fod->writedataactive) {
drivers/nvme/target/fc.c nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue) · line 705
Quoted 10 lines
drivers/nvme/target/fc.c +2
-1fod->aborted = true;spin_unlock (&fod->flock); tgtport->ops->fcp_abort ( &tgtport->fc_target_port, fod->fcpreq); - } + }else +spin_unlock (&fod->flock); } }
'writedataactive' is now unused, and should be removed.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 15/29 nvmet-fc: track hostport handle for associations
In reply to James SmartIn preparation for sending LS requests for an association that terminates, save and track the hosthandle that is part of the LS's that are received to create associations.
Support consists of:
- Create a hostport structure that will be 1:1 mapped to a host port handle. The hostport structure is specific to a targetport.
- Whenever an association is created, create a host port for the hosthandle the Create Association LS was received from. There will be only 1 hostport structure created, with all associations that have the same hosthandle sharing the hostport structure.
- When the association is terminated, the hostport reference will be removed. After the last association for the host port is removed, the hostport will be deleted.
- Add support for the new nvmet_fc_invalidate_host() interface. In the past, the LLDD didn't notify loss of connectivity to host ports - the LLD would simply reject new requests and wait for the kato timeout to kill the association. Now, when host port connectivity is lost, the LLDD can notify the transport. The transport will initiate the termination of all associations for that host port. When the last association has been terminated and the hosthandle will no longer be referenced, the new host_release callback will be made to the lldd.
- For compatibility with prior behavior which didn't report the
hosthandle: the LLDD must set hosthandle to NULL. In these cases, not LS request will be made, and no host_release callbacks will be made either.
--- drivers/nvme/target/fc.c | 177 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 170 insertions(+), 7 deletions(-)
drivers/nvme/target/fc.c +170 -7
@@ -33,6 +33,7 @@ struct nvmet_fc_ls_iod {struct nvmet_fc_tgtport *tgtport;struct nvmet_fc_tgt_assoc *assoc; +void *hosthandle;union nvmefc_ls_requests *rqstbuf;union nvmefc_ls_responses *rspbuf;
@@ -81,7 +82,6 @@ struct nvmet_fc_fcp_iod { };struct nvmet_fc_tgtport { -struct nvmet_fc_target_port fc_target_port;struct list_head tgt_list;/* nvmet_fc_target_list */
@@ -93,6 +93,7 @@ struct nvmet_fc_tgtport {struct list_head ls_list;struct list_head ls_busylist;struct list_head assoc_list; +struct list_head host_list;struct ida assoc_cnt;struct nvmet_fc_port_entry *pe;struct kref ref;
@@ -134,14 +135,24 @@ struct nvmet_fc_tgt_queue {struct nvmet_fc_fcp_iod fod[];/* array of fcp_iods */ }__aligned(sizeof(unsigned long long)) ; +struct nvmet_fc_hostport { +struct nvmet_fc_tgtport *tgtport; +void *hosthandle; +struct list_head host_list; +struct kref ref; +u8 invalid; +}; +struct nvmet_fc_tgt_assoc {u64 association_id;u32 a_id;struct nvmet_fc_tgtport *tgtport; +struct nvmet_fc_hostport *hostport;struct list_head a_list;struct nvmet_fc_tgt_queue *queues[NVMET_NR_QUEUES +1 ];struct kref ref;struct work_struct del_work; +atomic_t del_work_active; };
@@ -774,17 +785,114 @@ nvmet_fc_find_target_queue(struct nvmet_fc_tgtport *tgtport, }static void +nvmet_fc_hostport_free (struct kref *ref) +{ +struct nvmet_fc_hostport *hostport = + container_of(ref, struct nvmet_fc_hostport, ref); +struct nvmet_fc_tgtport *tgtport = hostport->tgtport; +unsigned long flags; + +spin_lock_irqsave (&tgtport->lock, flags); +list_del (&hostport->host_list); +spin_unlock_irqrestore (&tgtport->lock, flags); +if (tgtport->ops->host_release && hostport->invalid) + tgtport->ops->host_release (hostport->hosthandle); +kfree (hostport); +nvmet_fc_tgtport_put (tgtport); +} + +static void +nvmet_fc_hostport_put (struct nvmet_fc_hostport *hostport) +{ +kref_put (&hostport->ref, nvmet_fc_hostport_free); +} + +static int +nvmet_fc_hostport_get (struct nvmet_fc_hostport *hostport) +{ +return kref_get_unless_zero (&hostport->ref); +} + +static void +nvmet_fc_free_hostport (struct nvmet_fc_hostport *hostport) +{ +/* if LLDD not implemented, leave as NULL */ +if (!hostport->hosthandle) +return ; + +nvmet_fc_hostport_put (hostport); +} + +static struct nvmet_fc_hostport * +nvmet_fc_alloc_hostport (struct nvmet_fc_tgtport *tgtport,void *hosthandle) +{ +struct nvmet_fc_hostport *newhost, *host, *match =NULL ; +unsigned long flags; + +/* if LLDD not implemented, leave as NULL */ +if (!hosthandle) +return NULL ; + +/* take reference for what will be the newly allocated hostport */ +if (!nvmet_fc_tgtport_get (tgtport)) +return ERR_PTR (-EINVAL ); + + newhost =kzalloc (sizeof (*newhost),GFP_KERNEL ); +if (!newhost) { +spin_lock_irqsave (&tgtport->lock, flags); +list_for_each_entry (host, &tgtport->host_list, host_list) { +if (host->hosthandle == hosthandle && !host->invalid) { +if (nvmet_fc_hostport_get (host)) { + match = host; +break ; + } + } + } +spin_unlock_irqrestore (&tgtport->lock, flags); +/* no allocation - release reference */ +nvmet_fc_tgtport_put (tgtport); +return (match) ? match :ERR_PTR (-ENOMEM ); + } + + newhost->tgtport = tgtport; + newhost->hosthandle = hosthandle; +INIT_LIST_HEAD (&newhost->host_list); +kref_init (&newhost->ref); + +spin_lock_irqsave (&tgtport->lock, flags); +list_for_each_entry (host, &tgtport->host_list, host_list) { +if (host->hosthandle == hosthandle && !host->invalid) { +if (nvmet_fc_hostport_get (host)) { + match = host; +break ; + } + } + } +if (match) { +kfree (newhost); + newhost =NULL ; +/* releasing allocation - release reference */ +nvmet_fc_tgtport_put (tgtport); + }else +list_add_tail (&newhost->host_list, &tgtport->host_list); +spin_unlock_irqrestore (&tgtport->lock, flags); + +return (match) ? match : newhost; +} + +static void nvmet_fc_delete_assoc (struct work_struct *work) {struct nvmet_fc_tgt_assoc *assoc = container_of(work, struct nvmet_fc_tgt_assoc, del_work);nvmet_fc_delete_target_assoc (assoc); +atomic_set (&assoc->del_work_active,0 );nvmet_fc_tgt_a_put (assoc); }static struct nvmet_fc_tgt_assoc * -nvmet_fc_alloc_target_assoc (struct nvmet_fc_tgtport *tgtport) +nvmet_fc_alloc_target_assoc (struct nvmet_fc_tgtport *tgtport,void *hosthandle) {struct nvmet_fc_tgt_assoc *assoc, *tmpassoc;unsigned long flags;
@@ -801,13 +909,18 @@ nvmet_fc_alloc_target_assoc(struct nvmet_fc_tgtport *tgtport) goto out_free_assoc;if (!nvmet_fc_tgtport_get (tgtport)) - goto out_ida_put; + goto out_ida; + + assoc->hostport =nvmet_fc_alloc_hostport (tgtport, hosthandle); +if (IS_ERR (assoc->hostport)) + goto out_put; assoc->tgtport = tgtport; assoc->a_id = idx;INIT_LIST_HEAD (&assoc->a_list);kref_init (&assoc->ref);INIT_WORK (&assoc->del_work, nvmet_fc_delete_assoc); +atomic_set (&assoc->del_work_active,0 );while (needrandom) {get_random_bytes (&ran,sizeof (ran) -BYTES_FOR_QID );
@@ -829,7 +942,9 @@ nvmet_fc_alloc_target_assoc(struct nvmet_fc_tgtport *tgtport)return assoc; -out_ida_put: +out_put: +nvmet_fc_tgtport_put (tgtport); +out_ida:ida_simple_remove (&tgtport->assoc_cnt, idx); out_free_assoc:kfree (assoc);
@@ -844,6 +959,7 @@ nvmet_fc_target_assoc_free(struct kref *ref)struct nvmet_fc_tgtport *tgtport = assoc->tgtport;unsigned long flags; +nvmet_fc_free_hostport (assoc->hostport);spin_lock_irqsave (&tgtport->lock, flags);list_del (&assoc->a_list);spin_unlock_irqrestore (&tgtport->lock, flags);
@@ -1057,6 +1173,7 @@ nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo,INIT_LIST_HEAD (&newrec->ls_list);INIT_LIST_HEAD (&newrec->ls_busylist);INIT_LIST_HEAD (&newrec->assoc_list); +INIT_LIST_HEAD (&newrec->host_list);kref_init (&newrec->ref);ida_init (&newrec->assoc_cnt); newrec->max_sg_cnt = template->max_sgl_segments;
@@ -1133,14 +1250,21 @@ __nvmet_fc_free_assocs(struct nvmet_fc_tgtport *tgtport) {struct nvmet_fc_tgt_assoc *assoc, *next;unsigned long flags; +int ret;spin_lock_irqsave (&tgtport->lock, flags);list_for_each_entry_safe (assoc, next, &tgtport->assoc_list, a_list) {if (!nvmet_fc_tgt_a_get (assoc))continue ; -if (!schedule_work (&assoc->del_work)) + ret =atomic_cmpxchg (&assoc->del_work_active,0 ,1 ); +if (ret ==0 ) { +if (!schedule_work (&assoc->del_work)) +nvmet_fc_tgt_a_put (assoc); + }else { +/* already deleting - release local reference */ nvmet_fc_tgt_a_put (assoc); + } }spin_unlock_irqrestore (&tgtport->lock, flags); }
@@ -1178,6 +1302,36 @@ void nvmet_fc_invalidate_host(struct nvmet_fc_target_port *target_port, void *hosthandle) { + struct nvmet_fc_tgtport *tgtport = targetport_to_tgtport(target_port); +struct nvmet_fc_tgt_assoc *assoc, *next; +unsigned long flags; +bool noassoc = true; +int ret; + +spin_lock_irqsave (&tgtport->lock, flags); +list_for_each_entry_safe (assoc, next, + &tgtport->assoc_list, a_list) { +if (!assoc->hostport || + assoc->hostport->hosthandle != hosthandle) +continue ; +if (!nvmet_fc_tgt_a_get (assoc)) +continue ; + assoc->hostport->invalid =1 ; + noassoc = false; + ret =atomic_cmpxchg (&assoc->del_work_active,0 ,1 ); +if (ret ==0 ) { +if (!schedule_work (&assoc->del_work)) +nvmet_fc_tgt_a_put (assoc); + }else { +/* already deleting - release local reference */ +nvmet_fc_tgt_a_put (assoc); + } + } +spin_unlock_irqrestore (&tgtport->lock, flags); + +/* if there's nothing to wait for - call the callback */ +if (noassoc && tgtport->ops->host_release) + tgtport->ops->host_release (hosthandle); }EXPORT_SYMBOL_GPL (nvmet_fc_invalidate_host);
@@ -1192,6 +1346,7 @@ nvmet_fc_delete_ctrl(struct nvmet_ctrl *ctrl)struct nvmet_fc_tgt_queue *queue;unsigned long flags;bool found_ctrl = false; +int ret;/* this is a bit ugly, but don't want to make locks layered */ spin_lock_irqsave (&nvmet_fc_tgtlock, flags);
@@ -1215,8 +1370,14 @@ nvmet_fc_delete_ctrl(struct nvmet_ctrl *ctrl)nvmet_fc_tgtport_put (tgtport);if (found_ctrl) { -if (!schedule_work (&assoc->del_work)) + ret =atomic_cmpxchg (&assoc->del_work_active,0 ,1 ); +if (ret ==0 ) { +if (!schedule_work (&assoc->del_work)) +nvmet_fc_tgt_a_put (assoc); + }else { +/* already deleting - release local reference */ nvmet_fc_tgt_a_put (assoc); + }return ; }
@@ -1293,7 +1454,8 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport,else {/* new association w/ admin queue */ - iod->assoc =nvmet_fc_alloc_target_assoc (tgtport); + iod->assoc =nvmet_fc_alloc_target_assoc ( + tgtport, iod->hosthandle);if (!iod->assoc) ret =VERR_ASSOC_ALLOC_FAIL ;else {
@@ -1628,6 +1790,7 @@ nvmet_fc_rcv_ls_req(struct nvmet_fc_target_port *target_port, iod->fcpreq =NULL ;memcpy (iod->rqstbuf, lsreqbuf, lsreqbuf_len); iod->rqstdatalen = lsreqbuf_len; + iod->hosthandle = hosthandle;schedule_work (&iod->work);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 15/29] nvmet-fc: track hostport handle for associations
Quoted 30 lines · James Smart
In preparation for sending LS requests for an association that terminates, save and track the hosthandle that is part of the LS's that are received to create associations.
Support consists of:
- Create a hostport structure that will be 1:1 mapped to a host port handle. The hostport structure is specific to a targetport.
- Whenever an association is created, create a host port for the hosthandle the Create Association LS was received from. There will be only 1 hostport structure created, with all associations that have the same hosthandle sharing the hostport structure.
- When the association is terminated, the hostport reference will be removed. After the last association for the host port is removed, the hostport will be deleted.
- Add support for the newnvmet_fc_invalidate_host()interface. In the past, the LLDD didn't notify loss of connectivity to host ports - the LLD would simply reject new requests and wait for the kato timeout to kill the association. Now, when host port connectivity is lost, the LLDD can notify the transport. The transport will initiate the termination of all associations for that host port. When the last association has been terminated and the hosthandle will no longer be referenced, the newhost_releasecallback will be made to the lldd.
- For compatibility with prior behavior which didn't report the
hosthandle: the LLDD must set hosthandle to NULL. In these cases, not LS request will be made, and nohost_releasecallbacks will be made either.
Quoted 5 lines
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 177 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 170 insertions(+), 7 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 16/29 nvmet-fc: rename ls_list to ls_rcv_list
In reply to James SmartIn preparation to add ls request support, rename the current ls_list, which is RCV LS request only, to ls_rcv_list.
--- drivers/nvme/target/fc.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
drivers/nvme/target/fc.c +11 -11
@@ -29,7 +29,7 @@ struct nvmet_fc_ls_iod {struct nvmefc_ls_rsp *lsrsp;struct nvmefc_tgt_fcp_req *fcpreq;/* only if RS */ -struct list_head ls_list;/* tgtport->ls_list */ +struct list_head ls_rcv_list;/* tgtport->ls_rcv_list */ struct nvmet_fc_tgtport *tgtport;struct nvmet_fc_tgt_assoc *assoc;
@@ -90,7 +90,7 @@ struct nvmet_fc_tgtport {struct nvmet_fc_ls_iod *iod;spinlock_t lock; -struct list_head ls_list; +struct list_head ls_rcv_list;struct list_head ls_busylist;struct list_head assoc_list;struct list_head host_list;
@@ -346,7 +346,7 @@ nvmet_fc_alloc_ls_iodlist(struct nvmet_fc_tgtport *tgtport)for (i =0 ; i <NVMET_LS_CTX_COUNT ; iod++, i++) {INIT_WORK (&iod->work, nvmet_fc_handle_ls_rqst_work); iod->tgtport = tgtport; -list_add_tail (&iod->ls_list, &tgtport->ls_list); +list_add_tail (&iod->ls_rcv_list, &tgtport->ls_rcv_list); iod->rqstbuf = kzalloc(sizeof(union nvmefc_ls_requests) + sizeof(union nvmefc_ls_responses),
@@ -367,12 +367,12 @@ nvmet_fc_alloc_ls_iodlist(struct nvmet_fc_tgtport *tgtport) out_fail:kfree (iod->rqstbuf); -list_del (&iod->ls_list); +list_del (&iod->ls_rcv_list);for (iod--, i--; i >=0 ; iod--, i--) {fc_dma_unmap_single (tgtport->dev, iod->rspdma,sizeof (*iod->rspbuf),DMA_TO_DEVICE );kfree (iod->rqstbuf); -list_del (&iod->ls_list); +list_del (&iod->ls_rcv_list); }kfree (iod);
@@ -391,7 +391,7 @@ nvmet_fc_free_ls_iodlist(struct nvmet_fc_tgtport *tgtport) iod->rspdma,sizeof (*iod->rspbuf), DMA_TO_DEVICE);kfree (iod->rqstbuf); -list_del (&iod->ls_list); +list_del (&iod->ls_rcv_list); }kfree (tgtport->iod); }
@@ -403,10 +403,10 @@ nvmet_fc_alloc_ls_iod(struct nvmet_fc_tgtport *tgtport)unsigned long flags;spin_lock_irqsave (&tgtport->lock, flags); - iod =list_first_entry_or_null (&tgtport->ls_list, - struct nvmet_fc_ls_iod, ls_list); + iod =list_first_entry_or_null (&tgtport->ls_rcv_list, + struct nvmet_fc_ls_iod, ls_rcv_list);if (iod) -list_move_tail (&iod->ls_list, &tgtport->ls_busylist); +list_move_tail (&iod->ls_rcv_list, &tgtport->ls_busylist);spin_unlock_irqrestore (&tgtport->lock, flags);return iod; }
@@ -419,7 +419,7 @@ nvmet_fc_free_ls_iod(struct nvmet_fc_tgtport *tgtport,unsigned long flags;spin_lock_irqsave (&tgtport->lock, flags); -list_move (&iod->ls_list, &tgtport->ls_list); +list_move (&iod->ls_rcv_list, &tgtport->ls_rcv_list);spin_unlock_irqrestore (&tgtport->lock, flags); }
@@ -1170,7 +1170,7 @@ nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo, newrec->dev = dev; newrec->ops = template;spin_lock_init (&newrec->lock); -INIT_LIST_HEAD (&newrec->ls_list); +INIT_LIST_HEAD (&newrec->ls_rcv_list);INIT_LIST_HEAD (&newrec->ls_busylist);INIT_LIST_HEAD (&newrec->assoc_list);INIT_LIST_HEAD (&newrec->host_list);
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 16/29] nvmet-fc: rename ls_list to ls_rcv_list
Quoted 8 lines · James Smart
In preparation to add ls request support, rename the current
ls_list, which is RCV LS request only, tols_rcv_list.Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 17/29 nvmet-fc: Add Disconnect Association Xmt support
In reply to James SmartAs part of FC-NVME-2 (and ammendment on FC-NVME), the target is to send a Disconnect LS after an association is terminated and any exchanges for the association have been ABTS'd. The target is also not to send the receipt to any Disconnect Association LS, received to initiate the association termination or received while the association is terminating, until the Disconnect LS has been transmit.
Add support for sending Disconnect Association LS after all I/O's complete (which is after ABTS'd certainly). Utilizes the new LLDD api to send ls requests.
There is no need to track the Disconnect LS response or to retry after timeout. All spec requirements will have been met by waiting for i/o completion to initiate the transmission.
Add support for tracking the reception of Disconnect Association and defering the response transmission until after the Disconnect Association LS has been transmit.
--- drivers/nvme/target/fc.c | 298 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 287 insertions(+), 11 deletions(-)
drivers/nvme/target/fc.c +287 -11
@@ -25,7 +25,7 @@struct nvmet_fc_tgtport ;struct nvmet_fc_tgt_assoc ; -struct nvmet_fc_ls_iod { +struct nvmet_fc_ls_iod {// for an LS RQST RCV struct nvmefc_ls_rsp *lsrsp;struct nvmefc_tgt_fcp_req *fcpreq;/* only if RS */
@@ -45,6 +45,18 @@ struct nvmet_fc_ls_iod {struct work_struct work; }__aligned(sizeof(unsigned long long)) ; +struct nvmet_fc_ls_req_op {// for an LS RQST XMT +struct nvmefc_ls_req ls_req; + +struct nvmet_fc_tgtport *tgtport; +void *hosthandle; + +int ls_error; +struct list_head lsreq_list;/* tgtport->ls_req_list */ +bool req_queued; +}; + +/* desired maximum for a single sequence - if sg list allows it */ #define NVMET_FC_MAX_SEQ_LENGTH (256 * 1024)
@@ -91,6 +103,7 @@ struct nvmet_fc_tgtport {struct nvmet_fc_ls_iod *iod;spinlock_t lock;struct list_head ls_rcv_list; +struct list_head ls_req_list;struct list_head ls_busylist;struct list_head assoc_list;struct list_head host_list;
@@ -146,8 +159,10 @@ struct nvmet_fc_hostport {struct nvmet_fc_tgt_assoc {u64 association_id;u32 a_id; +atomic_t terminating;struct nvmet_fc_tgtport *tgtport;struct nvmet_fc_hostport *hostport; +struct nvmet_fc_ls_iod *rcv_disconn;struct list_head a_list;struct nvmet_fc_tgt_queue *queues[NVMET_NR_QUEUES +1 ];struct kref ref;
@@ -236,6 +251,8 @@ static int nvmet_fc_tgtport_get(struct nvmet_fc_tgtport *tgtport);static void nvmet_fc_handle_fcp_rqst (struct nvmet_fc_tgtport *tgtport,struct nvmet_fc_fcp_iod *fod);static void nvmet_fc_delete_target_assoc (struct nvmet_fc_tgt_assoc *assoc); +static void nvmet_fc_xmt_ls_rsp (struct nvmet_fc_tgtport *tgtport, +struct nvmet_fc_ls_iod *iod);/* *********************** FC-NVME DMA Handling **************************** */
@@ -327,6 +344,188 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, } +/* ********************** FC-NVME LS XMT Handling ************************* */ + + +static void +__nvmet_fc_finish_ls_req (struct nvmet_fc_ls_req_op *lsop) +{ +struct nvmet_fc_tgtport *tgtport = lsop->tgtport; +struct nvmefc_ls_req *lsreq = &lsop->ls_req; +unsigned long flags; + +spin_lock_irqsave (&tgtport->lock, flags); + +if (!lsop->req_queued) { +spin_unlock_irqrestore (&tgtport->lock, flags); +return ; + } + +list_del (&lsop->lsreq_list); + + lsop->req_queued = false; + +spin_unlock_irqrestore (&tgtport->lock, flags); + +fc_dma_unmap_single (tgtport->dev, lsreq->rqstdma, + (lsreq->rqstlen + lsreq->rsplen), +DMA_BIDIRECTIONAL ); + +nvmet_fc_tgtport_put (tgtport); +} + +static int +__nvmet_fc_send_ls_req (struct nvmet_fc_tgtport *tgtport, +struct nvmet_fc_ls_req_op *lsop, +void (*done)(struct nvmefc_ls_req *req,int status)) +{ +struct nvmefc_ls_req *lsreq = &lsop->ls_req; +unsigned long flags; +int ret =0 ; + +if (!tgtport->ops->ls_req) +return -EOPNOTSUPP ; + +if (!nvmet_fc_tgtport_get (tgtport)) +return -ESHUTDOWN ; + + lsreq->done = done; + lsop->req_queued = false; +INIT_LIST_HEAD (&lsop->lsreq_list); + + lsreq->rqstdma =fc_dma_map_single (tgtport->dev, lsreq->rqstaddr, + lsreq->rqstlen + lsreq->rsplen, +DMA_BIDIRECTIONAL ); +if (fc_dma_mapping_error (tgtport->dev, lsreq->rqstdma)) { + ret = -EFAULT ; + goto out_puttgtport; + } + lsreq->rspdma = lsreq->rqstdma + lsreq->rqstlen; + +spin_lock_irqsave (&tgtport->lock, flags); + +list_add_tail (&lsop->lsreq_list, &tgtport->ls_req_list); + + lsop->req_queued = true; + +spin_unlock_irqrestore (&tgtport->lock, flags); + + ret = tgtport->ops->ls_req (&tgtport->fc_target_port, lsop->hosthandle, + lsreq); +if (ret) + goto out_unlink; + +return 0 ; + +out_unlink: + lsop->ls_error = ret; +spin_lock_irqsave (&tgtport->lock, flags); + lsop->req_queued = false; +list_del (&lsop->lsreq_list); +spin_unlock_irqrestore (&tgtport->lock, flags); +fc_dma_unmap_single (tgtport->dev, lsreq->rqstdma, + (lsreq->rqstlen + lsreq->rsplen), +DMA_BIDIRECTIONAL ); +out_puttgtport: +nvmet_fc_tgtport_put (tgtport); + +return ret; +} + +static int +nvmet_fc_send_ls_req_async (struct nvmet_fc_tgtport *tgtport, +struct nvmet_fc_ls_req_op *lsop, +void (*done)(struct nvmefc_ls_req *req,int status)) +{ +/* don't wait for completion */ + +return __nvmet_fc_send_ls_req (tgtport, lsop, done); +} + +static void +nvmet_fc_disconnect_assoc_done (struct nvmefc_ls_req *lsreq,int status) +{ +struct nvmet_fc_ls_req_op *lsop = + container_of(lsreq, struct nvmet_fc_ls_req_op, ls_req); + +__nvmet_fc_finish_ls_req (lsop); + +/* fc-nvme target doesn't care about success or failure of cmd */ + +kfree (lsop); +} + +/* +* This routine sends a FC-NVME LS to disconnect (aka terminate) +* the FC-NVME Association. Terminating the association also +* terminates the FC-NVME connections (per queue, both admin and io +* queues) that are part of the association. E.g. things are torn +* down, and the related FC-NVME Association ID and Connection IDs +* become invalid. +* +* The behavior of the fc-nvme target is such that it's +* understanding of the association and connections will implicitly +* be torn down. The action is implicit as it may be due to a loss of +* connectivity with the fc-nvme host, so the target may never get a +* response even if it tried. As such, the action of this routine +* is to asynchronously send the LS, ignore any results of the LS, and +* continue on with terminating the association. If the fc-nvme host +* is present and receives the LS, it too can tear down. +*/ +static void +nvmet_fc_xmt_disconnect_assoc (struct nvmet_fc_tgt_assoc *assoc) +{ +struct nvmet_fc_tgtport *tgtport = assoc->tgtport; +struct fcnvme_ls_disconnect_assoc_rqst *discon_rqst; +struct fcnvme_ls_disconnect_assoc_acc *discon_acc; +struct nvmet_fc_ls_req_op *lsop; +struct nvmefc_ls_req *lsreq; +int ret; + +/* +* If ls_req is NULL or no hosthandle, it's an older lldd and no +* message is normal. Otherwise, send unless the hostport has +* already been invalidated by the lldd. +*/ +if (!tgtport->ops->ls_req || !assoc->hostport || + assoc->hostport->invalid) +return ; + + lsop =kzalloc ((sizeof (*lsop) + +sizeof (*discon_rqst) +sizeof (*discon_acc) + + tgtport->ops->lsrqst_priv_sz),GFP_KERNEL ); +if (!lsop) { +dev_info (tgtport->dev, +"{%d:%d} send Disconnect Association failed: ENOMEM\n" , + tgtport->fc_target_port.port_num, assoc->a_id); +return ; + } + + discon_rqst = (struct fcnvme_ls_disconnect_assoc_rqst *)&lsop[1 ]; + discon_acc = (struct fcnvme_ls_disconnect_assoc_acc *)&discon_rqst[1 ]; + lsreq = &lsop->ls_req; +if (tgtport->ops->lsrqst_priv_sz) + lsreq->private = (void *)&discon_acc[1 ]; +else + lsreq->private =NULL ; + + lsop->tgtport = tgtport; + lsop->hosthandle = assoc->hostport->hosthandle; + +nvmefc_fmt_lsreq_discon_assoc (lsreq, discon_rqst, discon_acc, + assoc->association_id); + + ret =nvmet_fc_send_ls_req_async (tgtport, lsop, + nvmet_fc_disconnect_assoc_done); +if (ret) { +dev_info (tgtport->dev, +"{%d:%d} XMT Disconnect Association failed: %d\n" , + tgtport->fc_target_port.port_num, assoc->a_id, ret); +kfree (lsop); + } +} + +/* *********************** FC-NVME Port Management ************************ */
@@ -689,10 +888,14 @@ nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue)struct nvmet_fc_defer_fcp_req *deferfcp, *tempptr;unsigned long flags;int i, writedataactive; -bool disconnect; +int disconnect; disconnect =atomic_xchg (&queue->connected,0 ); +/* if not connected, nothing to do */ +if (!disconnect) +return ; +spin_lock_irqsave (&queue->qlock, flags);/* abort outstanding io's */ for (i =0 ; i < queue->sqsize; fod++, i++) {
@@ -921,6 +1124,7 @@ nvmet_fc_alloc_target_assoc(struct nvmet_fc_tgtport *tgtport, void *hosthandle)kref_init (&assoc->ref);INIT_WORK (&assoc->del_work, nvmet_fc_delete_assoc);atomic_set (&assoc->del_work_active,0 ); +atomic_set (&assoc->terminating,0 );while (needrandom) {get_random_bytes (&ran,sizeof (ran) -BYTES_FOR_QID );
@@ -957,13 +1161,24 @@ nvmet_fc_target_assoc_free(struct kref *ref)struct nvmet_fc_tgt_assoc *assoc = container_of(ref, struct nvmet_fc_tgt_assoc, ref);struct nvmet_fc_tgtport *tgtport = assoc->tgtport; +struct nvmet_fc_ls_iod *oldls;unsigned long flags; +/* Send Disconnect now that all i/o has completed */ +nvmet_fc_xmt_disconnect_assoc (assoc); +nvmet_fc_free_hostport (assoc->hostport);spin_lock_irqsave (&tgtport->lock, flags);list_del (&assoc->a_list); + oldls = assoc->rcv_disconn;spin_unlock_irqrestore (&tgtport->lock, flags); +/* if pending Rcv Disconnect Association LS, send rsp now */ +if (oldls) +nvmet_fc_xmt_ls_rsp (tgtport, oldls);ida_simple_remove (&tgtport->assoc_cnt, assoc->a_id); +dev_info (tgtport->dev, +"{%d:%d} Association freed\n" , + tgtport->fc_target_port.port_num, assoc->a_id);kfree (assoc);nvmet_fc_tgtport_put (tgtport); }
@@ -986,7 +1201,13 @@ nvmet_fc_delete_target_assoc(struct nvmet_fc_tgt_assoc *assoc)struct nvmet_fc_tgtport *tgtport = assoc->tgtport;struct nvmet_fc_tgt_queue *queue;unsigned long flags; -int i; +int i, terminating; + + terminating =atomic_xchg (&assoc->terminating,1 ); + +/* if already terminating, do nothing */ +if (terminating) +return ;spin_lock_irqsave (&tgtport->lock, flags);for (i =NVMET_NR_QUEUES ; i >=0 ; i--) {
@@ -1002,6 +1223,10 @@ nvmet_fc_delete_target_assoc(struct nvmet_fc_tgt_assoc *assoc) }spin_unlock_irqrestore (&tgtport->lock, flags); +dev_info (tgtport->dev, +"{%d:%d} Association deleted\n" , + tgtport->fc_target_port.port_num, assoc->a_id); +nvmet_fc_tgt_a_put (assoc); }
@@ -1171,6 +1396,7 @@ nvmet_fc_register_targetport(struct nvmet_fc_port_info *pinfo, newrec->ops = template;spin_lock_init (&newrec->lock);INIT_LIST_HEAD (&newrec->ls_rcv_list); +INIT_LIST_HEAD (&newrec->ls_req_list);INIT_LIST_HEAD (&newrec->ls_busylist);INIT_LIST_HEAD (&newrec->assoc_list);INIT_LIST_HEAD (&newrec->host_list);
@@ -1407,6 +1633,13 @@ nvmet_fc_unregister_targetport(struct nvmet_fc_target_port *target_port)/* terminate any outstanding associations */ __nvmet_fc_free_assocs (tgtport); +/* +* should terminate LS's as well. However, LS's will be generated +* at the tail end of association termination, so they likely don't +* exist yet. And even if they did, it's worthwhile to just let +* them finish and targetport ref counting will clean things up. +*/ +nvmet_fc_tgtport_put (tgtport);return 0 ;
@@ -1414,7 +1647,7 @@ nvmet_fc_unregister_targetport(struct nvmet_fc_target_port *target_port)EXPORT_SYMBOL_GPL (nvmet_fc_unregister_targetport); -/* *********************** FC-NVME LS Handling **************************** */ +/* ********************** FC-NVME LS RCV Handling ************************* */ static void
@@ -1481,6 +1714,10 @@ nvmet_fc_ls_create_association(struct nvmet_fc_tgtport *tgtport,atomic_set (&queue->connected,1 ); queue->sqhd =0 ;/* best place to init value */ +dev_info (tgtport->dev, +"{%d:%d} Association created\n" , + tgtport->fc_target_port.port_num, iod->assoc->a_id); +/* format a response */ iod->lsrsp->rsplen =sizeof (*acc);
@@ -1588,7 +1825,11 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport, be16_to_cpu(rqst->connect_cmd.qid))); } -static void +/* +* Returns true if the LS response is to be transmit +* Returns false if the LS response is to be delayed +*/ +static int nvmet_fc_ls_disconnect (struct nvmet_fc_tgtport *tgtport,struct nvmet_fc_ls_iod *iod) {
@@ -1597,13 +1838,15 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport,struct fcnvme_ls_disconnect_assoc_acc *acc = &iod->rspbuf->rsp_dis_assoc;struct nvmet_fc_tgt_assoc *assoc; +struct nvmet_fc_ls_iod *oldls =NULL ; +unsigned long flags;int ret =0 ;memset (acc,0 ,sizeof (*acc)); ret =nvmefc_vldt_lsreq_discon_assoc (iod->rqstdatalen, rqst);if (!ret) { -/* match an active association */ +/* match an active association - takes an assoc ref if !NULL */ assoc =nvmet_fc_find_target_assoc (tgtport,be64_to_cpu (rqst->associd.association_id)); iod->assoc = assoc;
@@ -1621,7 +1864,7 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport, FCNVME_RJT_RC_INV_ASSOC :FCNVME_RJT_RC_LOGIC , FCNVME_RJT_EXP_NONE, 0); -return ; +return true; }/* format a response */
@@ -1634,9 +1877,40 @@ nvmet_fc_ls_disconnect(struct nvmet_fc_tgtport *tgtport, FCNVME_LS_DISCONNECT_ASSOC);/* release get taken in nvmet_fc_find_target_assoc */ -nvmet_fc_tgt_a_put (iod->assoc); +nvmet_fc_tgt_a_put (assoc); + +/* +* The rules for LS response says the response cannot +* go back until ABTS's have been sent for all outstanding +* I/O and a Disconnect Association LS has been sent. +* So... save off the Disconnect LS to send the response +* later. If there was a prior LS already saved, replace +* it with the newer one and send a can't perform reject +* on the older one. +*/ +spin_lock_irqsave (&tgtport->lock, flags); + oldls = assoc->rcv_disconn; + assoc->rcv_disconn = iod; +spin_unlock_irqrestore (&tgtport->lock, flags); -nvmet_fc_delete_target_assoc (iod->assoc); +nvmet_fc_delete_target_assoc (assoc); + +if (oldls) { +dev_info (tgtport->dev, +"{%d:%d} Multiple Disconnect Association LS's " +"received\n" , + tgtport->fc_target_port.port_num, assoc->a_id); +/* overwrite good response with bogus failure */ + oldls->lsrsp->rsplen =nvme_fc_format_rjt (oldls->rspbuf, +sizeof (*iod->rspbuf), +/* ok to use rqst, LS is same */ + rqst->w0.ls_cmd, +FCNVME_RJT_RC_UNAB , +FCNVME_RJT_EXP_NONE ,0 ); +nvmet_fc_xmt_ls_rsp (tgtport, oldls); + } + +return false; }
@@ -1681,6 +1955,7 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport, struct nvmet_fc_ls_iod *iod) {struct fcnvme_ls_rqst_w0 *w0 = &iod->rqstbuf->rq_cr_assoc.w0; +bool sendrsp = true; iod->lsrsp->nvme_fc_private = iod; iod->lsrsp->rspbuf = iod->rspbuf;
@@ -1707,7 +1982,7 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport,break ;case FCNVME_LS_DISCONNECT_ASSOC :/* Terminate a Queue/Connection or the Association */ -nvmet_fc_ls_disconnect (tgtport, iod); + sendrsp =nvmet_fc_ls_disconnect (tgtport, iod);break ;default : iod->lsrsp->rsplen = nvme_fc_format_rjt(iod->rspbuf,
@@ -1715,7 +1990,8 @@ nvmet_fc_handle_ls_rqst(struct nvmet_fc_tgtport *tgtport, FCNVME_RJT_RC_INVAL, FCNVME_RJT_EXP_NONE, 0); } -nvmet_fc_xmt_ls_rsp (tgtport, iod); +if (sendrsp) +nvmet_fc_xmt_ls_rsp (tgtport, iod); }/*
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 17/29] nvmet-fc: Add Disconnect Association Xmt support
Quoted 46 lines · James Smart
As part of FC-NVME-2 (and ammendment on FC-NVME), the target is to send a Disconnect LS after an association is terminated and any exchanges for the association have been ABTS'd. The target is also not to send the receipt to any Disconnect Association LS, received to initiate the association termination or received while the association is terminating, until the Disconnect LS has been transmit.
Add support for sending Disconnect Association LS after all I/O's complete (which is after ABTS'd certainly). Utilizes the new LLDD api to send ls requests.
There is no need to track the Disconnect LS response or to retry after timeout. All spec requirements will have been met by waiting for i/o completion to initiate the transmission.
Add support for tracking the reception of Disconnect Association and defering the response transmission until after the Disconnect Association LS has been transmit.
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fc.c | 298 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 287 insertions(+), 11 deletions(-)drivers/nvme/target/fc.c +6
-1@@ -25,7 +25,7 @@struct nvmet_fc_tgtport ;struct nvmet_fc_tgt_assoc ; -struct nvmet_fc_ls_iod { +struct nvmet_fc_ls_iod {// for an LS RQST RCV struct nvmefc_ls_rsp *lsrsp;struct nvmefc_tgt_fcp_req *fcpreq;/* only if RS */ @@ -45,6 +45,18 @@ struct nvmet_fc_ls_iod {struct work_struct work; }__aligned(sizeof(unsigned long long)) ; +struct nvmet_fc_ls_req_op {// for an LS RQST XMT +struct nvmefc_ls_req ls_req; + +struct nvmet_fc_tgtport *tgtport; +void *hosthandle;
drivers/nvme/target/fc.c struct nvmet_fc_ls_iod { · line 62
Quoted 10 lines
drivers/nvme/target/fc.c +7
-0+ +int ls_error; +struct list_head lsreq_list;/* tgtport->ls_req_list */ +bool req_queued; +}; + +/* desired maximum for a single sequence - if sg list allows it */ #define NVMET_FC_MAX_SEQ_LENGTH (256 * 1024)
Please, use normal comments, not C++-style // thingies.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 18/29 nvme-fcloop: refactor to enable target to host LS
In reply to James SmartCurrently nvmefc-loop only sends LS's from host to target. Slightly rework data structures and routine names to reflect this path. Allows a straight-forward conversion to be used by ls's from target to host.
--- drivers/nvme/target/fcloop.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
drivers/nvme/target/fcloop.c +13 -6
@@ -226,7 +226,13 @@ struct fcloop_nport {u32 port_id; }; +enum { +H2T =0 , +T2H =1 , +}; +struct fcloop_lsreq { +int lsdir;/* H2T or T2H */ struct nvmefc_ls_req *lsreq;struct nvmefc_ls_rsp ls_rsp;int status;
@@ -323,7 +329,7 @@ fcloop_rport_lsrqst_work(struct work_struct *work) }static int -fcloop_ls_req (struct nvme_fc_local_port *localport, +fcloop_h2t_ls_req (struct nvme_fc_local_port *localport,struct nvme_fc_remote_port *remoteport,struct nvmefc_ls_req *lsreq) {
@@ -331,6 +337,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,struct fcloop_rport *rport = remoteport->private;int ret =0 ; + tls_req->lsdir =H2T ; tls_req->lsreq = lsreq;INIT_LIST_HEAD (&tls_req->ls_list);
@@ -351,7 +358,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, }static int -fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *targetport, +fcloop_h2t_xmt_ls_rsp (struct nvmet_fc_target_port *targetport,struct nvmefc_ls_rsp *lsrsp) {struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);
@@ -762,7 +769,7 @@ fcloop_fcp_req_release(struct nvmet_fc_target_port *tgtport, }static void -fcloop_ls_abort (struct nvme_fc_local_port *localport, +fcloop_h2t_ls_abort (struct nvme_fc_local_port *localport,struct nvme_fc_remote_port *remoteport,struct nvmefc_ls_req *lsreq) {
@@ -880,9 +887,9 @@ static struct nvme_fc_port_template fctemplate = { .remoteport_delete = fcloop_remoteport_delete, .create_queue = fcloop_create_queue, .delete_queue = fcloop_delete_queue, - .ls_req = fcloop_ls_req, + .ls_req = fcloop_h2t_ls_req, .fcp_io = fcloop_fcp_req, - .ls_abort = fcloop_ls_abort, + .ls_abort = fcloop_h2t_ls_abort, .fcp_abort = fcloop_fcp_abort, .max_hw_queues = FCLOOP_HW_QUEUES, .max_sgl_segments = FCLOOP_SGL_SEGS,
@@ -897,7 +904,7 @@ static struct nvme_fc_port_template fctemplate = {static struct nvmet_fc_target_template tgttemplate = { .targetport_delete = fcloop_targetport_delete, - .xmt_ls_rsp = fcloop_xmt_ls_rsp, + .xmt_ls_rsp = fcloop_h2t_xmt_ls_rsp, .fcp_op = fcloop_fcp_op, .fcp_abort = fcloop_tgt_fcp_abort, .fcp_req_release = fcloop_fcp_req_release,
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 18/29] nvme-fcloop: refactor to enable target to host LS
Quoted 18 lines · James Smart
Currently nvmefc-loop only sends LS's from host to target. Slightly rework data structures and routine names to reflect this path. Allows a straight-forward conversion to be used by ls's from target to host.
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fcloop.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)drivers/nvme/target/fcloop.c +0
-0@@ -226,7 +226,13 @@ struct fcloop_nport {u32 port_id; };
drivers/nvme/target/fcloop.c struct fcloop_nport { · line 238
Quoted 10 lines
drivers/nvme/target/fcloop.c +6
-0+enum { +H2T =0 , +T2H =1 , +}; +struct fcloop_lsreq { +int lsdir;/* H2T or T2H */ struct nvmefc_ls_req *lsreq;struct nvmefc_ls_rsp ls_rsp;int status;
Please move it after 'ls_rsp'; otherwise we'll have a misalignment on 64 bit.
Quoted 49 lines
drivers/nvme/target/fcloop.c +6
-5@@ -323,7 +329,7 @@ fcloop_rport_lsrqst_work(struct work_struct *work) }static int -fcloop_ls_req (struct nvme_fc_local_port *localport, +fcloop_h2t_ls_req (struct nvme_fc_local_port *localport,struct nvme_fc_remote_port *remoteport,struct nvmefc_ls_req *lsreq) {@@ -331,6 +337,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport,struct fcloop_rport *rport = remoteport->private;int ret =0 ; + tls_req->lsdir =H2T ; tls_req->lsreq = lsreq;INIT_LIST_HEAD (&tls_req->ls_list);@@ -351,7 +358,7 @@ fcloop_ls_req(struct nvme_fc_local_port *localport, }static int -fcloop_xmt_ls_rsp (struct nvmet_fc_target_port *targetport, +fcloop_h2t_xmt_ls_rsp (struct nvmet_fc_target_port *targetport,struct nvmefc_ls_rsp *lsrsp) {struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp);@@ -762,7 +769,7 @@ fcloop_fcp_req_release(struct nvmet_fc_target_port *tgtport, }static void -fcloop_ls_abort (struct nvme_fc_local_port *localport, +fcloop_h2t_ls_abort (struct nvme_fc_local_port *localport,struct nvme_fc_remote_port *remoteport,struct nvmefc_ls_req *lsreq) {@@ -880,9 +887,9 @@ static struct nvme_fc_port_template fctemplate = { .remoteport_delete = fcloop_remoteport_delete, .create_queue = fcloop_create_queue, .delete_queue = fcloop_delete_queue, - .ls_req = fcloop_ls_req, + .ls_req = fcloop_h2t_ls_req, .fcp_io = fcloop_fcp_req, - .ls_abort = fcloop_ls_abort, + .ls_abort = fcloop_h2t_ls_abort, .fcp_abort = fcloop_fcp_abort, .max_hw_queues = FCLOOP_HW_QUEUES, .max_sgl_segments = FCLOOP_SGL_SEGS,
drivers/nvme/target/fcloop.c static struct nvme_fc_port_template fctemplate = { · line 910
Quoted 8 lines
drivers/nvme/target/fcloop.c +1
-1@@ -897,7 +904,7 @@ static struct nvme_fc_port_template fctemplate = {static struct nvmet_fc_target_template tgttemplate = { .targetport_delete = fcloop_targetport_delete, - .xmt_ls_rsp = fcloop_xmt_ls_rsp, + .xmt_ls_rsp = fcloop_h2t_xmt_ls_rsp, .fcp_op = fcloop_fcp_op, .fcp_abort = fcloop_tgt_fcp_abort, .fcp_req_release = fcloop_fcp_req_release,
Otherwise:
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 19/29 nvme-fcloop: add target to host LS request support
In reply to James SmartAdd support for performing LS requests from target to host. Include sending request from targetport, reception into host, host sending ls rsp.
--- drivers/nvme/target/fcloop.c | 131 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 118 insertions(+), 13 deletions(-)
drivers/nvme/target/fcloop.c +118 -13
@@ -208,10 +208,13 @@ struct fcloop_rport { };struct fcloop_tport { -struct nvmet_fc_target_port *targetport; -struct nvme_fc_remote_port *remoteport; -struct fcloop_nport *nport; -struct fcloop_lport *lport; +struct nvmet_fc_target_port *targetport; +struct nvme_fc_remote_port *remoteport; +struct fcloop_nport *nport; +struct fcloop_lport *lport; +spinlock_t lock; +struct list_head ls_list; +struct work_struct ls_work; };struct fcloop_nport {
@@ -226,13 +229,7 @@ struct fcloop_nport {u32 port_id; }; -enum { -H2T =0 , -T2H =1 , -}; -struct fcloop_lsreq { -int lsdir;/* H2T or T2H */ struct nvmefc_ls_req *lsreq;struct nvmefc_ls_rsp ls_rsp;int status;
@@ -337,7 +334,6 @@ fcloop_h2t_ls_req(struct nvme_fc_local_port *localport,struct fcloop_rport *rport = remoteport->private;int ret =0 ; - tls_req->lsdir =H2T ; tls_req->lsreq = lsreq;INIT_LIST_HEAD (&tls_req->ls_list);
@@ -351,8 +347,9 @@ fcloop_h2t_ls_req(struct nvme_fc_local_port *localport, } tls_req->status =0 ; - ret =nvmet_fc_rcv_ls_req (rport->targetport,NULL , &tls_req->ls_rsp, - lsreq->rqstaddr, lsreq->rqstlen); + ret =nvmet_fc_rcv_ls_req (rport->targetport, rport, + &tls_req->ls_rsp, + lsreq->rqstaddr, lsreq->rqstlen);return ret; }
@@ -384,6 +381,99 @@ fcloop_h2t_xmt_ls_rsp(struct nvmet_fc_target_port *targetport,return 0 ; } +static void +fcloop_tport_lsrqst_work (struct work_struct *work) +{ +struct fcloop_tport *tport = + container_of(work, struct fcloop_tport, ls_work); +struct fcloop_lsreq *tls_req; + +spin_lock (&tport->lock); +for (;;) { + tls_req =list_first_entry_or_null (&tport->ls_list, + struct fcloop_lsreq, ls_list); +if (!tls_req) +break ; + +list_del (&tls_req->ls_list); +spin_unlock (&tport->lock); + + tls_req->lsreq->done (tls_req->lsreq, tls_req->status); +/* +* callee may free memory containing tls_req. +* do not reference lsreq after this. +*/ + +spin_lock (&tport->lock); + } +spin_unlock (&tport->lock); +} + +static int +fcloop_t2h_ls_req (struct nvmet_fc_target_port *targetport,void *hosthandle, +struct nvmefc_ls_req *lsreq) +{ +struct fcloop_lsreq *tls_req = lsreq->private; +struct fcloop_tport *tport = targetport->private; +int ret =0 ; + +/* +* hosthandle should be the dst.rport value. +* hosthandle ignored as fcloop currently is +* 1:1 tgtport vs remoteport +*/ + tls_req->lsreq = lsreq; +INIT_LIST_HEAD (&tls_req->ls_list); + +if (!tport->remoteport) { + tls_req->status = -ECONNREFUSED ; +spin_lock (&tport->lock); +list_add_tail (&tport->ls_list, &tls_req->ls_list); +spin_unlock (&tport->lock); +schedule_work (&tport->ls_work); +return ret; + } + + tls_req->status =0 ; + ret =nvme_fc_rcv_ls_req (tport->remoteport, &tls_req->ls_rsp, + lsreq->rqstaddr, lsreq->rqstlen); + +return ret; +} + +static int +fcloop_t2h_xmt_ls_rsp (struct nvme_fc_local_port *localport, +struct nvme_fc_remote_port *remoteport, +struct nvmefc_ls_rsp *lsrsp) +{ +struct fcloop_lsreq *tls_req =ls_rsp_to_lsreq (lsrsp); +struct nvmefc_ls_req *lsreq = tls_req->lsreq; +struct fcloop_rport *rport = remoteport->private; +struct nvmet_fc_target_port *targetport = rport->targetport; +struct fcloop_tport *tport; + +memcpy (lsreq->rspaddr, lsrsp->rspbuf, + ((lsreq->rsplen < lsrsp->rsplen) ? + lsreq->rsplen : lsrsp->rsplen)); + lsrsp->done (lsrsp); + +if (targetport) { + tport = targetport->private; +spin_lock (&tport->lock); +list_add_tail (&tport->ls_list, &tls_req->ls_list); +spin_unlock (&tport->lock); +schedule_work (&tport->ls_work); + } + +return 0 ; +} + +static void +fcloop_t2h_host_release (void *hosthandle) +{ +/* host handle ignored for now */ +} +/* * Simulate reception of RSCN and converting it to a initiator transport * call to rescan a remote port.
@@ -776,6 +866,12 @@ fcloop_h2t_ls_abort(struct nvme_fc_local_port *localport, }static void +fcloop_t2h_ls_abort (struct nvmet_fc_target_port *targetport, +void *hosthandle,struct nvmefc_ls_req *lsreq) +{ +} + +static void fcloop_fcp_abort(struct nvme_fc_local_port *localport, struct nvme_fc_remote_port *remoteport, void *hw_queue_handle,
@@ -874,6 +970,7 @@ fcloop_targetport_delete(struct nvmet_fc_target_port *targetport) {struct fcloop_tport *tport = targetport->private; +flush_work (&tport->ls_work);fcloop_nport_put (tport->nport); }
@@ -891,6 +988,7 @@ static struct nvme_fc_port_template fctemplate = { .fcp_io = fcloop_fcp_req, .ls_abort = fcloop_h2t_ls_abort, .fcp_abort = fcloop_fcp_abort, + .xmt_ls_rsp = fcloop_t2h_xmt_ls_rsp, .max_hw_queues = FCLOOP_HW_QUEUES, .max_sgl_segments = FCLOOP_SGL_SEGS, .max_dif_sgl_segments = FCLOOP_SGL_SEGS,
@@ -909,6 +1007,9 @@ static struct nvmet_fc_target_template tgttemplate = { .fcp_abort = fcloop_tgt_fcp_abort, .fcp_req_release = fcloop_fcp_req_release, .discovery_event = fcloop_tgt_discovery_evt, + .ls_req = fcloop_t2h_ls_req, + .ls_abort = fcloop_t2h_ls_abort, + .host_release = fcloop_t2h_host_release, .max_hw_queues = FCLOOP_HW_QUEUES, .max_sgl_segments = FCLOOP_SGL_SEGS, .max_dif_sgl_segments = FCLOOP_SGL_SEGS,
@@ -917,6 +1018,7 @@ static struct nvmet_fc_target_template tgttemplate = { .target_features = 0, /* sizes of additional private data for data structures */ .target_priv_sz = sizeof(struct fcloop_tport), + .lsrqst_priv_sz = sizeof(struct fcloop_lsreq), }; static ssize_t
@@ -1266,6 +1368,9 @@ fcloop_create_target_port(struct device *dev, struct device_attribute *attr, tport->nport = nport; tport->lport = nport->lport; nport->tport = tport; +spin_lock_init (&tport->lock); +INIT_WORK (&tport->ls_work, fcloop_tport_lsrqst_work); +INIT_LIST_HEAD (&tport->ls_list);return count; }
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 19/29] nvme-fcloop: add target to host LS request support
Quoted 9 lines · James Smart
Add support for performing LS requests from target to host. Include sending request from targetport, reception into host, host sending ls rsp.
Signed-off-by James Smart <[email protected]>--- drivers/nvme/target/fcloop.c | 131 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 118 insertions(+), 13 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 20/29 lpfc: Refactor lpfc nvme headers
In reply to James SmartA lot of files in lpfc include nvme headers, building up relationships that require a file to change for its headers when there is no other change necessary. It would be better to localize the nvme headers.
There is also no need for separate nvme (initiator) and nvmet (tgt) header files.
Refactor the inclusion of nvme headers so that all nvme items are included by lpfc_nvme.h
Merge lpfc_nvmet.h into lpfc_nvme.h so that there is a single header used by both the nvme and nvmet sides. This prepares for structure sharing between the two roles. Prep to add shared function prototypes for upcoming shared routines.
--- drivers/scsi/lpfc/lpfc_attr.c | 3 - drivers/scsi/lpfc/lpfc_ct.c | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 3 - drivers/scsi/lpfc/lpfc_hbadisc.c | 2 - drivers/scsi/lpfc/lpfc_init.c | 3 - drivers/scsi/lpfc/lpfc_mem.c | 4 - drivers/scsi/lpfc/lpfc_nportdisc.c | 2 - drivers/scsi/lpfc/lpfc_nvme.c | 3 - drivers/scsi/lpfc/lpfc_nvme.h | 147 ++++++++++++++++++++++++++++++++++ drivers/scsi/lpfc/lpfc_nvmet.c | 5 -- drivers/scsi/lpfc/lpfc_nvmet.h | 158 ------------------------------------- drivers/scsi/lpfc/lpfc_sli.c | 3 - 12 files changed, 147 insertions(+), 187 deletions(-) delete mode 100644 drivers/scsi/lpfc/lpfc_nvmet.h
drivers/scsi/lpfc/lpfc_attr.c +0 -3
@@ -37,8 +37,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
@@ -48,7 +46,6 @@#include "lpfc.h" #include "lpfc_scsi.h" #include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_logmsg.h" #include "lpfc_version.h" #include "lpfc_compat.h"
drivers/scsi/lpfc/lpfc_ct.c +0 -1
@@ -44,7 +44,6 @@#include "lpfc_disc.h" #include "lpfc.h" #include "lpfc_scsi.h" -#include "lpfc_nvme.h" #include "lpfc_logmsg.h" #include "lpfc_crtn.h" #include "lpfc_version.h"
drivers/scsi/lpfc/lpfc_debugfs.c +0 -3
@@ -39,8 +39,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
@@ -50,7 +48,6 @@#include "lpfc.h" #include "lpfc_scsi.h" #include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_logmsg.h" #include "lpfc_crtn.h" #include "lpfc_vport.h"
drivers/scsi/lpfc/lpfc_hbadisc.c +0 -2
@@ -35,8 +35,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_nl.h"
drivers/scsi/lpfc/lpfc_init.c +0 -3
@@ -50,8 +50,6 @@#include <scsi/scsi_tcq.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
@@ -61,7 +59,6 @@#include "lpfc.h" #include "lpfc_scsi.h" #include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_logmsg.h" #include "lpfc_crtn.h" #include "lpfc_vport.h"
drivers/scsi/lpfc/lpfc_mem.c +0 -4
@@ -31,8 +31,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
@@ -41,8 +39,6 @@#include "lpfc_disc.h" #include "lpfc.h" #include "lpfc_scsi.h" -#include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_crtn.h" #include "lpfc_logmsg.h"
drivers/scsi/lpfc/lpfc_nportdisc.c +0 -2
@@ -32,8 +32,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
drivers/scsi/lpfc/lpfc_nvme.c +0 -3
@@ -36,9 +36,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme.h> -#include <linux/nvme-fc-driver.h> -#include <linux/nvme-fc.h> #include "lpfc_version.h" #include "lpfc_hw4.h" #include "lpfc_hw.h"
drivers/scsi/lpfc/lpfc_nvme.h +147 -0
156 lines not on this page · raw message
drivers/scsi/lpfc/lpfc_nvmet.c +0 -5
@@ -36,10 +36,6 @@#include <scsi/scsi_transport_fc.h> #include <scsi/fc/fc_fs.h> -#include <linux/nvme.h> -#include <linux/nvme-fc-driver.h> -#include <linux/nvme-fc.h> -#include "lpfc_version.h" #include "lpfc_hw4.h" #include "lpfc_hw.h"
@@ -50,7 +46,6 @@#include "lpfc.h" #include "lpfc_scsi.h" #include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_logmsg.h" #include "lpfc_crtn.h" #include "lpfc_vport.h"
drivers/scsi/lpfc/lpfc_nvmet.h deleted +0 -158
158 lines not on this page · raw message
drivers/scsi/lpfc/lpfc_sli.c +0 -3
@@ -39,8 +39,6 @@#include <asm/set_memory.h> #endif -#include <linux/nvme-fc-driver.h> -#include "lpfc_hw4.h" #include "lpfc_hw.h" #include "lpfc_sli.h"
@@ -50,7 +48,6 @@#include "lpfc.h" #include "lpfc_scsi.h" #include "lpfc_nvme.h" -#include "lpfc_nvmet.h" #include "lpfc_crtn.h" #include "lpfc_logmsg.h" #include "lpfc_compat.h"
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 20/29] lpfc: Refactor lpfc nvme headers
Quoted 17 lines · James Smart
A lot of files in lpfc include nvme headers, building up relationships that require a file to change for its headers when there is no other change necessary. It would be better to localize the nvme headers.
There is also no need for separate nvme (initiator) and nvmet (tgt) header files.
Refactor the inclusion of nvme headers so that all nvme items are included by
lpfc_nvme.hMerge
lpfc_nvmet.hintolpfc_nvme.hso that there is a single header used by both the nvme and nvmet sides. This prepares for structure sharing between the two roles. Prep to add shared function prototypes for upcoming shared routines.Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>
Quoted 16 lines
--- drivers/scsi/lpfc/lpfc_attr.c | 3 - drivers/scsi/lpfc/lpfc_ct.c | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 3 - drivers/scsi/lpfc/lpfc_hbadisc.c | 2 - drivers/scsi/lpfc/lpfc_init.c | 3 - drivers/scsi/lpfc/lpfc_mem.c | 4 - drivers/scsi/lpfc/lpfc_nportdisc.c | 2 - drivers/scsi/lpfc/lpfc_nvme.c | 3 - drivers/scsi/lpfc/lpfc_nvme.h | 147 ++++++++++++++++++++++++++++++++++ drivers/scsi/lpfc/lpfc_nvmet.c | 5 -- drivers/scsi/lpfc/lpfc_nvmet.h | 158 ------------------------------------- drivers/scsi/lpfc/lpfc_sli.c | 3 - 12 files changed, 147 insertions(+), 187 deletions(-) delete mode 100644 drivers/scsi/lpfc/lpfc_nvmet.h
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 21/29 lpfc: Refactor nvmet_rcv_ctx to create lpfc_async_xchg_ctx
In reply to James SmartTo support FC-NVME-2 support (actually FC-NVME (rev 1) with Ammendment 1), both the nvme (host) and nvmet (controller/target) sides will need to be able to receive LS requests. Currently, this support is in the nvmet side only. To prepare for both sides supporting LS receive, rename lpfc_nvmet_rcv_ctx to lpfc_async_xchg_ctx and commonize the definition.
--- drivers/scsi/lpfc/lpfc.h | 2 +- drivers/scsi/lpfc/lpfc_crtn.h | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 2 +- drivers/scsi/lpfc/lpfc_init.c | 2 +- drivers/scsi/lpfc/lpfc_nvme.h | 7 +-- drivers/scsi/lpfc/lpfc_nvmet.c | 109 ++++++++++++++++++++------------------- drivers/scsi/lpfc/lpfc_sli.c | 2 +- 7 files changed, 63 insertions(+), 62 deletions(-)
drivers/scsi/lpfc/lpfc.h +1 -1
@@ -143,7 +143,7 @@ struct lpfc_dmabuf {struct lpfc_nvmet_ctxbuf {struct list_head list; -struct lpfc_nvmet_rcv_ctx *context; +struct lpfc_async_xchg_ctx *context;struct lpfc_iocbq *iocbq;struct lpfc_sglq *sglq;struct work_struct defer_work;
drivers/scsi/lpfc/lpfc_crtn.h +0 -1
@@ -24,7 +24,6 @@ typedef int (*node_filter)(struct lpfc_nodelist *, void *);struct fc_rport ;struct fc_frame_header ; -struct lpfc_nvmet_rcv_ctx ;void lpfc_down_link (struct lpfc_hba *,LPFC_MBOXQ_t *);void lpfc_sli_read_link_ste (struct lpfc_hba *);void lpfc_dump_mem (struct lpfc_hba *,LPFC_MBOXQ_t *,uint16_t ,uint16_t );
drivers/scsi/lpfc/lpfc_debugfs.c +1 -1
@@ -1032,7 +1032,7 @@ lpfc_debugfs_nvmestat_data(struct lpfc_vport *vport, char *buf, int size) {struct lpfc_hba *phba = vport->phba;struct lpfc_nvmet_tgtport *tgtp; -struct lpfc_nvmet_rcv_ctx *ctxp, *next_ctxp; +struct lpfc_async_xchg_ctx *ctxp, *next_ctxp;struct nvme_fc_local_port *localport;struct lpfc_fc4_ctrl_stat *cstat;struct lpfc_nvme_lport *lport;
drivers/scsi/lpfc/lpfc_init.c +1 -1
@@ -1038,7 +1038,7 @@ static int lpfc_hba_down_post_s4(struct lpfc_hba *phba) {struct lpfc_io_buf *psb, *psb_next; -struct lpfc_nvmet_rcv_ctx *ctxp, *ctxp_next; +struct lpfc_async_xchg_ctx *ctxp, *ctxp_next;struct lpfc_sli4_hdw_queue *qp;LIST_HEAD (aborts);LIST_HEAD (nvme_aborts);
drivers/scsi/lpfc/lpfc_nvme.h +4 -3
@@ -163,13 +163,14 @@ struct lpfc_nvmet_ctx_info {#define lpfc_get_ctx_list (phba, cpu, mrq) \ (phba->sli4_hba.nvmet_ctx_info + ((cpu * phba->cfg_nvmet_mrq) + mrq)) -struct lpfc_nvmet_rcv_ctx { +struct lpfc_async_xchg_ctx {union { -struct nvmefc_ls_rsp ls_rsp;struct nvmefc_tgt_fcp_req fcp_req; - } ctx; + } hdlrctx;struct list_head list;struct lpfc_hba *phba; +struct nvmefc_ls_req *ls_req; +struct nvmefc_ls_rsp ls_rsp;struct lpfc_iocbq *wqeq;struct lpfc_iocbq *abort_wqeq;spinlock_t ctxlock;/* protect flag access */
drivers/scsi/lpfc/lpfc_nvmet.c +55 -54
370 lines not on this page · raw message
drivers/scsi/lpfc/lpfc_sli.c +1 -1
@@ -19894,7 +19894,7 @@ lpfc_sli4_issue_wqe(struct lpfc_hba *phba, struct lpfc_sli4_hdw_queue *qp, struct lpfc_iocbq *pwqe) {union lpfc_wqe128 *wqe = &pwqe->wqe; -struct lpfc_nvmet_rcv_ctx *ctxp; +struct lpfc_async_xchg_ctx *ctxp;struct lpfc_queue *wq;struct lpfc_sglq *sglq;struct lpfc_sli_ring *pring;
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 21/29] lpfc: Refactor nvmet_rcv_ctx to create lpfc_async_xchg_ctx
Quoted 8 lines · James Smart
To support FC-NVME-2 support (actually FC-NVME (rev 1) with Ammendment 1), both the nvme (host) and nvmet (controller/target) sides will need to be able to receive LS requests. Currently, this support is in the nvmet side only. To prepare for both sides supporting LS receive, rename
lpfc_nvmet_rcv_ctxtolpfc_async_xchg_ctxand commonize the definition.Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>
Quoted 10 lines
--- drivers/scsi/lpfc/lpfc.h | 2 +- drivers/scsi/lpfc/lpfc_crtn.h | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 2 +- drivers/scsi/lpfc/lpfc_init.c | 2 +- drivers/scsi/lpfc/lpfc_nvme.h | 7 +-- drivers/scsi/lpfc/lpfc_nvmet.c | 109 ++++++++++++++++++++------------------- drivers/scsi/lpfc/lpfc_sli.c | 2 +- 7 files changed, 63 insertions(+), 62 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 22/29 lpfc: Commonize lpfc_async_xchg_ctx state and flag definitions
In reply to James SmartThe last step of commonization is to remove the 'T' suffix from state and flag field definitions. This is minor, but removes the mental association that it solely applies to nvmet use.
--- drivers/scsi/lpfc/lpfc_init.c | 2 +- drivers/scsi/lpfc/lpfc_nvme.h | 37 +++++----- drivers/scsi/lpfc/lpfc_nvmet.c | 158 ++++++++++++++++++++--------------------- 3 files changed, 100 insertions(+), 97 deletions(-)
drivers/scsi/lpfc/lpfc_init.c +1 -1
@@ -1105,7 +1105,7 @@ lpfc_hba_down_post_s4(struct lpfc_hba *phba) &nvmet_aborts);spin_unlock_irq (&phba->sli4_hba.abts_nvmet_buf_list_lock);list_for_each_entry_safe (ctxp, ctxp_next, &nvmet_aborts, list) { - ctxp->flag &= ~(LPFC_NVMET_XBUSY |LPFC_NVMET_ABORT_OP ); + ctxp->flag &= ~(LPFC_NVME_XBUSY |LPFC_NVME_ABORT_OP );lpfc_nvmet_ctxbuf_post (phba, ctxp->ctxbuf); } }
drivers/scsi/lpfc/lpfc_nvme.h +20 -17
@@ -163,6 +163,26 @@ struct lpfc_nvmet_ctx_info {#define lpfc_get_ctx_list (phba, cpu, mrq) \ (phba->sli4_hba.nvmet_ctx_info + ((cpu * phba->cfg_nvmet_mrq) + mrq)) +/* Values for state field of struct lpfc_async_xchg_ctx */ +#define LPFC_NVME_STE_LS_RCV 1 +#define LPFC_NVME_STE_LS_ABORT 2 +#define LPFC_NVME_STE_LS_RSP 3 +#define LPFC_NVME_STE_RCV 4 +#define LPFC_NVME_STE_DATA 5 +#define LPFC_NVME_STE_ABORT 6 +#define LPFC_NVME_STE_DONE 7 +#define LPFC_NVME_STE_FREE 0xff + +/* Values for flag field of struct lpfc_async_xchg_ctx */ +#define LPFC_NVME_IO_INP 0x1/* IO is in progress on exchange */ +#define LPFC_NVME_ABORT_OP 0x2/* Abort WQE issued on exchange */ +#define LPFC_NVME_XBUSY 0x4/* XB bit set on IO cmpl */ +#define LPFC_NVME_CTX_RLS 0x8/* ctx free requested */ +#define LPFC_NVME_ABTS_RCV 0x10/* ABTS received on exchange */ +#define LPFC_NVME_CTX_REUSE_WQ 0x20/* ctx reused via WQ */ +#define LPFC_NVME_DEFER_WQFULL 0x40/* Waiting on a free WQE */ +#define LPFC_NVME_TNOTIFY 0x80/* notify transport of abts */ +struct lpfc_async_xchg_ctx {union {struct nvmefc_tgt_fcp_req fcp_req;
@@ -182,24 +202,7 @@ struct lpfc_async_xchg_ctx {uint16_t cpu;uint16_t idx;uint16_t state; -/* States */ -#define LPFC_NVMET_STE_LS_RCV 1 -#define LPFC_NVMET_STE_LS_ABORT 2 -#define LPFC_NVMET_STE_LS_RSP 3 -#define LPFC_NVMET_STE_RCV 4 -#define LPFC_NVMET_STE_DATA 5 -#define LPFC_NVMET_STE_ABORT 6 -#define LPFC_NVMET_STE_DONE 7 -#define LPFC_NVMET_STE_FREE 0xffuint16_t flag; -#define LPFC_NVMET_IO_INP 0x1/* IO is in progress on exchange */ -#define LPFC_NVMET_ABORT_OP 0x2/* Abort WQE issued on exchange */ -#define LPFC_NVMET_XBUSY 0x4/* XB bit set on IO cmpl */ -#define LPFC_NVMET_CTX_RLS 0x8/* ctx free requested */ -#define LPFC_NVMET_ABTS_RCV 0x10/* ABTS received on exchange */ -#define LPFC_NVMET_CTX_REUSE_WQ 0x20/* ctx reused via WQ */ -#define LPFC_NVMET_DEFER_WQFULL 0x40/* Waiting on a free WQE */ -#define LPFC_NVMET_TNOTIFY 0x80/* notify transport of abts */ struct rqb_dmabuf *rqb_buffer;struct lpfc_nvmet_ctxbuf *ctxbuf;struct lpfc_sli4_hdw_queue *hdwq;
drivers/scsi/lpfc/lpfc_nvmet.c +79 -79
519 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 22/29] lpfc: Commonize lpfc_async_xchg_ctx state and flag definitions
Quoted 4 lines · James Smart
The last step of commonization is to remove the 'T' suffix from state and flag field definitions. This is minor, but removes the mental association that it solely applies to nvmet use.
Quoted 8 lines
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_init.c | 2 +- drivers/scsi/lpfc/lpfc_nvme.h | 37 +++++----- drivers/scsi/lpfc/lpfc_nvmet.c | 158 ++++++++++++++++++++--------------------- 3 files changed, 100 insertions(+), 97 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 23/29 lpfc: Refactor NVME LS receive handling
In reply to James SmartIn preparation for supporting both intiator mode and target mode receiving NVME LS's, commonize the existing NVME LS request receive handling found in the base driver and in the nvmet side.
Using the original lpfc_nvmet_unsol_ls_event() and lpfc_nvme_unsol_ls_buffer() routines as a templates, commonize the reception of an NVME LS request. The common routine will validate the LS request, that it was received from a logged-in node, and allocate a lpfc_async_xchg_ctx that is used to manage the LS request. The role of the port is then inspected to determine which handler is to receive the LS - nvme or nvmet. As such, the nvmet handler is tied back in. A handler is created in nvme and is stubbed out.
--- drivers/scsi/lpfc/lpfc_crtn.h | 6 +- drivers/scsi/lpfc/lpfc_nvme.c | 19 +++++ drivers/scsi/lpfc/lpfc_nvme.h | 5 ++ drivers/scsi/lpfc/lpfc_nvmet.c | 163 ++++++++++------------------------------- drivers/scsi/lpfc/lpfc_sli.c | 121 +++++++++++++++++++++++++++++- 5 files changed, 184 insertions(+), 130 deletions(-)
drivers/scsi/lpfc/lpfc_crtn.h +4 -2
@@ -564,8 +564,10 @@ void lpfc_nvme_update_localport(struct lpfc_vport *vport);int lpfc_nvmet_create_targetport (struct lpfc_hba *phba);int lpfc_nvmet_update_targetport (struct lpfc_hba *phba);void lpfc_nvmet_destroy_targetport (struct lpfc_hba *phba); -void lpfc_nvmet_unsol_ls_event (struct lpfc_hba *phba, -struct lpfc_sli_ring *pring,struct lpfc_iocbq *piocb); +int lpfc_nvme_handle_lsreq (struct lpfc_hba *phba, +struct lpfc_async_xchg_ctx *axchg); +int lpfc_nvmet_handle_lsreq (struct lpfc_hba *phba, +struct lpfc_async_xchg_ctx *axchg);void lpfc_nvmet_unsol_fcp_event (struct lpfc_hba *phba,uint32_t idx,struct rqb_dmabuf *nvmebuf,uint64_t isr_ts,uint8_t cqflag);
drivers/scsi/lpfc/lpfc_nvme.c +19 -0
@@ -391,6 +391,25 @@ lpfc_nvme_remoteport_delete(struct nvme_fc_remote_port *remoteport)return ; } +/** +* lpfc_nvme_handle_lsreq - Process an unsolicited NVME LS request +* @phba: pointer to lpfc hba data structure. +* @axchg: pointer to exchange context for the NVME LS request +* +* This routine is used for processing an asychronously received NVME LS +* request. Any remaining validation is done and the LS is then forwarded +* to the nvme-fc transport via nvme_fc_rcv_ls_req(). +* +* Returns 0 if LS was handled and delivered to the transport +* Returns 1 if LS failed to be handled and should be dropped +*/ +int +lpfc_nvme_handle_lsreq (struct lpfc_hba *phba, +struct lpfc_async_xchg_ctx *axchg) +{ +return 1 ; +} +static void lpfc_nvme_cmpl_gen_req (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe,struct lpfc_wcqe_complete *wcqe)
drivers/scsi/lpfc/lpfc_nvme.h +5 -0
@@ -189,6 +189,7 @@ struct lpfc_async_xchg_ctx { } hdlrctx;struct list_head list;struct lpfc_hba *phba; +struct lpfc_nodelist *ndlp;struct nvmefc_ls_req *ls_req;struct nvmefc_ls_rsp ls_rsp;struct lpfc_iocbq *wqeq;
@@ -203,6 +204,7 @@ struct lpfc_async_xchg_ctx {uint16_t idx;uint16_t state;uint16_t flag; +void *payload;struct rqb_dmabuf *rqb_buffer;struct lpfc_nvmet_ctxbuf *ctxbuf;struct lpfc_sli4_hdw_queue *hdwq;
@@ -225,3 +227,6 @@ struct lpfc_async_xchg_ctx {/* routines found in lpfc_nvme.c */ /* routines found in lpfc_nvmet.c */ +int lpfc_nvme_unsol_ls_issue_abort (struct lpfc_hba *phba, +struct lpfc_async_xchg_ctx *ctxp,uint32_t sid, +uint16_t xri);
drivers/scsi/lpfc/lpfc_nvmet.c +38 -125
216 lines not on this page · raw message
drivers/scsi/lpfc/lpfc_sli.c +118 -3
139 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 23/29] lpfc: Refactor NVME LS receive handling
Quoted 15 lines · James Smart
In preparation for supporting both intiator mode and target mode receiving NVME LS's, commonize the existing NVME LS request receive handling found in the base driver and in the nvmet side.
Using the original
lpfc_nvmet_unsol_ls_event()andlpfc_nvme_unsol_ls_buffer()routines as a templates, commonize the reception of an NVME LS request. The common routine will validate the LS request, that it was received from a logged-in node, and allocate alpfc_async_xchg_ctxthat is used to manage the LS request. The role of the port is then inspected to determine which handler is to receive the LS - nvme or nvmet. As such, the nvmet handler is tied back in. A handler is created in nvme and is stubbed out.Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>
Quoted 8 lines
--- drivers/scsi/lpfc/lpfc_crtn.h | 6 +- drivers/scsi/lpfc/lpfc_nvme.c | 19 +++++ drivers/scsi/lpfc/lpfc_nvme.h | 5 ++ drivers/scsi/lpfc/lpfc_nvmet.c | 163 ++++++++++------------------------------- drivers/scsi/lpfc/lpfc_sli.c | 121 +++++++++++++++++++++++++++++- 5 files changed, 184 insertions(+), 130 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 24/29 lpfc: Refactor Send LS Request support
In reply to James SmartCurrently, the ability to send an NVME LS request is limited to the nvme (host) side of the driver. In preparation of both the nvme and nvmet sides support Send LS Request, rework the existing send ls_req and ls_req completion routines such that there is common code that can be used by both sides.
--- drivers/scsi/lpfc/lpfc_nvme.c | 289 +++++++++++++++++++++++++----------------- drivers/scsi/lpfc/lpfc_nvme.h | 13 ++ 2 files changed, 184 insertions(+), 118 deletions(-)
drivers/scsi/lpfc/lpfc_nvme.c +171 -118
@@ -410,43 +410,43 @@ lpfc_nvme_handle_lsreq(struct lpfc_hba *phba,return 1 ; } -static void -lpfc_nvme_cmpl_gen_req (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe, -struct lpfc_wcqe_complete *wcqe) +/** +* __lpfc_nvme_ls_req_cmp - Generic completion handler for a NVME +* LS request. +* @phba: Pointer to HBA context object +* @vport: The local port that issued the LS +* @cmdwqe: Pointer to driver command WQE object. +* @wcqe: Pointer to driver response CQE object. +* +* This function is the generic completion handler for NVME LS requests. +* The function updates any states and statistics, calls the transport +* ls_req done() routine, then tears down the command and buffers used +* for the LS request. +**/ +void +__lpfc_nvme_ls_req_cmp (struct lpfc_hba *phba,struct lpfc_vport *vport, +struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe) { -struct lpfc_vport *vport = cmdwqe->vport; -struct lpfc_nvme_lport *lport; -uint32_t status;struct nvmefc_ls_req *pnvme_lsreq;struct lpfc_dmabuf *buf_ptr;struct lpfc_nodelist *ndlp; +uint32_t status; pnvme_lsreq = (struct nvmefc_ls_req *)cmdwqe->context2; + ndlp = (struct lpfc_nodelist *)cmdwqe->context1; status =bf_get (lpfc_wcqe_c_status, wcqe) &LPFC_IOCB_STATUS_MASK ; -if (vport->localport) { - lport = (struct lpfc_nvme_lport *)vport->localport->private; -if (lport) { -atomic_inc (&lport->fc4NvmeLsCmpls); -if (status) { -if (bf_get (lpfc_wcqe_c_xb, wcqe)) -atomic_inc (&lport->cmpl_ls_xb); -atomic_inc (&lport->cmpl_ls_err); - } - } - } - - ndlp = (struct lpfc_nodelist *)cmdwqe->context1;lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_DISC , -"6047 nvme cmpl Enter " -"Data %px DID %x Xri: %x status %x reason x%x " -"cmd:x%px lsreg:x%px bmp:x%px ndlp:x%px\n" , +"6047 NVMEx LS REQ %px cmpl DID %x Xri: %x " +"status %x reason x%x cmd:x%px lsreg:x%px bmp:x%px " +"ndlp:x%px\n" , pnvme_lsreq, ndlp ? ndlp->nlp_DID :0 , cmdwqe->sli4_xritag, status, (wcqe->parameter &0xffff ), cmdwqe, pnvme_lsreq, cmdwqe->context3, ndlp); -lpfc_nvmeio_data (phba,"NVME LS CMPL: xri x%x stat x%x parm x%x\n" , +lpfc_nvmeio_data (phba,"NVMEx LS CMPL: xri x%x stat x%x parm x%x\n" , cmdwqe->sli4_xritag, status, wcqe->parameter);if (cmdwqe->context3) {
@@ -459,7 +459,7 @@ lpfc_nvme_cmpl_gen_req(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe, pnvme_lsreq->done (pnvme_lsreq, status); else lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC, -"6046 nvme cmpl without done call back? " +"6046 NVMEx cmpl without done call back? " "Data %px DID %x Xri: %x status %x\n" , pnvme_lsreq, ndlp ? ndlp->nlp_DID :0 , cmdwqe->sli4_xritag, status);
@@ -470,6 +470,31 @@ lpfc_nvme_cmpl_gen_req(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe,lpfc_sli_release_iocbq (phba, cmdwqe); } +static void +lpfc_nvme_ls_req_cmp (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe) +{ +struct lpfc_vport *vport = cmdwqe->vport; +struct lpfc_nvme_lport *lport; +uint32_t status; + + status =bf_get (lpfc_wcqe_c_status, wcqe) &LPFC_IOCB_STATUS_MASK ; + +if (vport->localport) { + lport = (struct lpfc_nvme_lport *)vport->localport->private; +if (lport) { +atomic_inc (&lport->fc4NvmeLsCmpls); +if (status) { +if (bf_get (lpfc_wcqe_c_xb, wcqe)) +atomic_inc (&lport->cmpl_ls_xb); +atomic_inc (&lport->cmpl_ls_err); + } + } + } + +__lpfc_nvme_ls_req_cmp (phba, vport, cmdwqe, wcqe); +} + static int lpfc_nvme_gen_req(struct lpfc_vport *vport, struct lpfc_dmabuf *bmp, struct lpfc_dmabuf *inp,
@@ -571,13 +596,6 @@ lpfc_nvme_gen_req(struct lpfc_vport *vport, struct lpfc_dmabuf *bmp,/* Issue GEN REQ WQE for NPORT <did> */ -lpfc_printf_vlog (vport,KERN_INFO ,LOG_ELS , -"6050 Issue GEN REQ WQE to NPORT x%x " -"Data: x%x x%x wq:x%px lsreq:x%px bmp:x%px " -"xmit:%d 1st:%d\n" , - ndlp->nlp_DID, genwqe->iotag, - vport->port_state, - genwqe, pnvme_lsreq, bmp, xmit_len, first_len); genwqe->wqe_cmpl = cmpl; genwqe->iocb_cmpl =NULL ; genwqe->drvrTimeout = tmo +LPFC_DRVR_TIMEOUT ;
@@ -589,105 +607,108 @@ lpfc_nvme_gen_req(struct lpfc_vport *vport, struct lpfc_dmabuf *bmp, rc =lpfc_sli4_issue_wqe (phba, &phba->sli4_hba.hdwq[0 ], genwqe);if (rc) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_ELS , +lpfc_printf_vlog (vport,KERN_ERR ,LOG_NVME_DISC |LOG_ELS ,"6045 Issue GEN REQ WQE to NPORT x%x " -"Data: x%x x%x\n" , +"Data: x%x x%x rc x%x\n" , ndlp->nlp_DID, genwqe->iotag, - vport->port_state); + vport->port_state, rc);lpfc_sli_release_iocbq (phba, genwqe);return 1 ; } + +lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_DISC |LOG_ELS , +"6050 Issue GEN REQ WQE to NPORT x%x " +"Data: oxid: x%x state: x%x wq:x%px lsreq:x%px " +"bmp:x%px xmit:%d 1st:%d\n" , + ndlp->nlp_DID, genwqe->sli4_xritag, + vport->port_state, + genwqe, pnvme_lsreq, bmp, xmit_len, first_len);return 0 ; } +/** -* lpfc_nvme_ls_req - Issue an Link Service request -* @lpfc_pnvme: Pointer to the driver's nvme instance data -* @lpfc_nvme_lport: Pointer to the driver's local port data -* @lpfc_nvme_rport: Pointer to the rport getting the @lpfc_nvme_ereq +* __lpfc_nvme_ls_req - Generic service routine to issue an NVME LS request +* @vport: The local port issuing the LS +* @ndlp: The remote port to send the LS to +* @pnvme_lsreq: Pointer to LS request structure from the transport * -* Driver registers this routine to handle any link service request -* from the nvme_fc transport to a remote nvme-aware port. +* Routine validates the ndlp, builds buffers and sends a GEN_REQUEST +* WQE to perform the LS operation. * * Return value : * 0 - Success -* TODO: What are the failure codes. +* non-zero: various error codes, in form of -Exxx **/ -static int -lpfc_nvme_ls_req (struct nvme_fc_local_port *pnvme_lport, -struct nvme_fc_remote_port *pnvme_rport, -struct nvmefc_ls_req *pnvme_lsreq) +int +__lpfc_nvme_ls_req (struct lpfc_vport *vport,struct lpfc_nodelist *ndlp, +struct nvmefc_ls_req *pnvme_lsreq, +void (*gen_req_cmp)(struct lpfc_hba *phba, +struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe)) { -int ret =0 ; -struct lpfc_nvme_lport *lport; -struct lpfc_nvme_rport *rport; -struct lpfc_vport *vport; -struct lpfc_nodelist *ndlp; -struct ulp_bde64 *bpl;struct lpfc_dmabuf *bmp; +struct ulp_bde64 *bpl; +int ret;uint16_t ntype, nstate; -/* there are two dma buf in the request, actually there is one and -* the second one is just the start address + cmd size. -* Before calling lpfc_nvme_gen_req these buffers need to be wrapped -* in a lpfc_dmabuf struct. When freeing we just free the wrapper -* because the nvem layer owns the data bufs. -* We do not have to break these packets open, we don't care what is in -* them. And we do not have to look at the resonse data, we only care -* that we got a response. All of the caring is going to happen in the -* nvme-fc layer. -*/ - - lport = (struct lpfc_nvme_lport *)pnvme_lport->private; - rport = (struct lpfc_nvme_rport *)pnvme_rport->private; -if (unlikely (!lport) ||unlikely (!rport)) -return -EINVAL ; - - vport = lport->vport; - -if (vport->load_flag &FC_UNLOADING ) -return -ENODEV ; - -/* Need the ndlp. It is stored in the driver's rport. */ - ndlp = rport->ndlp;if (!ndlp || !NLP_CHK_NODE_ACT (ndlp)) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NODE |LOG_NVME_IOERR , -"6051 Remoteport x%px, rport has invalid ndlp. " -"Failing LS Req\n" , pnvme_rport); +lpfc_printf_vlog (vport,KERN_ERR , +LOG_NVME_DISC |LOG_NODE |LOG_NVME_IOERR , +"6051 NVMEx LS REQ: Bad NDLP x%px, Failing " +"LS Req\n" , + ndlp);return -ENODEV ; } -/* The remote node has to be a mapped nvme target or an -* unmapped nvme initiator or it's an error. -*/ ntype = ndlp->nlp_type; nstate = ndlp->nlp_state;if ((ntype &NLP_NVME_TARGET && nstate !=NLP_STE_MAPPED_NODE ) || (ntype &NLP_NVME_INITIATOR && nstate !=NLP_STE_UNMAPPED_NODE )) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NODE |LOG_NVME_IOERR , -"6088 DID x%06x not ready for " -"IO. State x%x, Type x%x\n" , - pnvme_rport->port_id, - ndlp->nlp_state, ndlp->nlp_type); +lpfc_printf_vlog (vport,KERN_ERR , +LOG_NVME_DISC |LOG_NODE |LOG_NVME_IOERR , +"6088 NVMEx LS REQ: Fail DID x%06x not " +"ready for IO. Type x%x, State x%x\n" , + ndlp->nlp_DID, ntype, nstate);return -ENODEV ; } - bmp =kmalloc (sizeof (struct lpfc_dmabuf ),GFP_KERNEL ); + +/* +* there are two dma buf in the request, actually there is one and +* the second one is just the start address + cmd size. +* Before calling lpfc_nvme_gen_req these buffers need to be wrapped +* in a lpfc_dmabuf struct. When freeing we just free the wrapper +* because the nvem layer owns the data bufs. +* We do not have to break these packets open, we don't care what is +* in them. And we do not have to look at the resonse data, we only +* care that we got a response. All of the caring is going to happen +* in the nvme-fc layer. +*/ + + bmp =kmalloc (sizeof (*bmp),GFP_KERNEL );if (!bmp) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NVME_DISC , -"6044 Could not find node for DID %x\n" , - pnvme_rport->port_id); -return 2 ; +lpfc_printf_vlog (vport,KERN_ERR , +LOG_NVME_DISC |LOG_NVME_IOERR , +"6044 NVMEx LS REQ: Could not alloc LS buf " +"for DID %x\n" , + ndlp->nlp_DID); +return -ENOMEM ; } -INIT_LIST_HEAD (&bmp->list); + bmp->virt =lpfc_mbuf_alloc (vport->phba,MEM_PRI , &(bmp->phys));if (!bmp->virt) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NVME_DISC , -"6042 Could not find node for DID %x\n" , - pnvme_rport->port_id); +lpfc_printf_vlog (vport,KERN_ERR , +LOG_NVME_DISC |LOG_NVME_IOERR , +"6042 NVMEx LS REQ: Could not alloc mbuf " +"for DID %x\n" , + ndlp->nlp_DID);kfree (bmp); -return 3 ; +return -ENOMEM ; } + +INIT_LIST_HEAD (&bmp->list); + bpl = (struct ulp_bde64 *)bmp->virt; bpl->addrHigh =le32_to_cpu (putPaddrHigh (pnvme_lsreq->rqstdma)); bpl->addrLow =le32_to_cpu (putPaddrLow (pnvme_lsreq->rqstdma));
@@ -702,37 +723,69 @@ lpfc_nvme_ls_req(struct nvme_fc_local_port *pnvme_lport, bpl->tus.f.bdeSize = pnvme_lsreq->rsplen; bpl->tus.w =le32_to_cpu (bpl->tus.w); -/* Expand print to include key fields. */ lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_DISC , -"6149 Issue LS Req to DID 0x%06x lport x%px, " -"rport x%px lsreq x%px rqstlen:%d rsplen:%d " -"%pad %pad\n" , - ndlp->nlp_DID, pnvme_lport, pnvme_rport, - pnvme_lsreq, pnvme_lsreq->rqstlen, - pnvme_lsreq->rsplen, &pnvme_lsreq->rqstdma, - &pnvme_lsreq->rspdma); - -atomic_inc (&lport->fc4NvmeLsRequests); +"6149 NVMEx LS REQ: Issue to DID 0x%06x lsreq x%px, " +"rqstlen:%d rsplen:%d %pad %pad\n" , + ndlp->nlp_DID, pnvme_lsreq, pnvme_lsreq->rqstlen, + pnvme_lsreq->rsplen, &pnvme_lsreq->rqstdma, + &pnvme_lsreq->rspdma); -/* Hardcode the wait to 30 seconds. Connections are failing otherwise. -* This code allows it all to work. -*/ ret =lpfc_nvme_gen_req (vport, bmp, pnvme_lsreq->rqstaddr, - pnvme_lsreq, lpfc_nvme_cmpl_gen_req, - ndlp,2 ,30 ,0 ); + pnvme_lsreq, gen_req_cmp, ndlp,2 , +LPFC_NVME_LS_TIMEOUT ,0 );if (ret !=WQE_SUCCESS ) { -atomic_inc (&lport->xmt_ls_err); -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NVME_DISC , -"6052 EXIT. issue ls wqe failed lport x%px, " -"rport x%px lsreq x%px Status %x DID %x\n" , - pnvme_lport, pnvme_rport, pnvme_lsreq, - ret, ndlp->nlp_DID); +lpfc_printf_vlog (vport,KERN_ERR , +LOG_NVME_DISC |LOG_NVME_IOERR , +"6052 NVMEx REQ: EXIT. issue ls wqe failed " +"lsreq x%px Status %x DID %x\n" , + pnvme_lsreq, ret, ndlp->nlp_DID);lpfc_mbuf_free (vport->phba, bmp->virt, bmp->phys);kfree (bmp); -return ret; +return -EIO ; } -/* Stub in routine and return 0 for now. */ +return 0 ; +} + +/** +* lpfc_nvme_ls_req - Issue an NVME Link Service request +* @lpfc_nvme_lport: Transport localport that LS is to be issued from. +* @lpfc_nvme_rport: Transport remoteport that LS is to be sent to. +* @pnvme_lsreq - the transport nvme_ls_req structure for the LS +* +* Driver registers this routine to handle any link service request +* from the nvme_fc transport to a remote nvme-aware port. +* +* Return value : +* 0 - Success +* non-zero: various error codes, in form of -Exxx +**/ +static int +lpfc_nvme_ls_req (struct nvme_fc_local_port *pnvme_lport, +struct nvme_fc_remote_port *pnvme_rport, +struct nvmefc_ls_req *pnvme_lsreq) +{ +struct lpfc_nvme_lport *lport; +struct lpfc_nvme_rport *rport; +struct lpfc_vport *vport; +int ret; + + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; + rport = (struct lpfc_nvme_rport *)pnvme_rport->private; +if (unlikely (!lport) ||unlikely (!rport)) +return -EINVAL ; + + vport = lport->vport; +if (vport->load_flag &FC_UNLOADING ) +return -ENODEV ; + +atomic_inc (&lport->fc4NvmeLsRequests); + + ret =__lpfc_nvme_ls_req (vport, rport->ndlp, pnvme_lsreq, + lpfc_nvme_ls_req_cmp); +if (ret) +atomic_inc (&lport->xmt_ls_err); +return ret; }
drivers/scsi/lpfc/lpfc_nvme.h +13 -0
@@ -80,6 +80,12 @@ struct lpfc_nvme_fcpreq_priv {struct lpfc_io_buf *nvme_buf; }; +/* +* set NVME LS request timeouts to 30s. It is larger than the 2*R_A_TOV +* set by the spec, which appears to have issues with some devices. +*/ +#define LPFC_NVME_LS_TIMEOUT 30 +#define LPFC_NVMET_DEFAULT_SEGS (64 + 1)/* 256K IOs */ #define LPFC_NVMET_RQE_MIN_POST 128
@@ -225,6 +231,13 @@ struct lpfc_async_xchg_ctx {/* routines found in lpfc_nvme.c */ +int __lpfc_nvme_ls_req (struct lpfc_vport *vport,struct lpfc_nodelist *ndlp, +struct nvmefc_ls_req *pnvme_lsreq, +void (*gen_req_cmp)(struct lpfc_hba *phba, +struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe)); +void __lpfc_nvme_ls_req_cmp (struct lpfc_hba *phba,struct lpfc_vport *vport, +struct lpfc_iocbq *cmdwqe,struct lpfc_wcqe_complete *wcqe);/* routines found in lpfc_nvmet.c */ int lpfc_nvme_unsol_ls_issue_abort(struct lpfc_hba *phba,
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 24/29] lpfc: Refactor Send LS Request support
Quoted 6 lines · James Smart
Currently, the ability to send an NVME LS request is limited to the nvme (host) side of the driver. In preparation of both the nvme and nvmet sides support Send LS Request, rework the existing send
ls_reqandls_reqcompletion routines such that there is common code that can be used by both sides.
Quoted 7 lines
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvme.c | 289 +++++++++++++++++++++++++----------------- drivers/scsi/lpfc/lpfc_nvme.h | 13 ++ 2 files changed, 184 insertions(+), 118 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 25/29 lpfc: Refactor Send LS Abort support
In reply to James SmartSend LS Abort support is needed when Send LS Request is supported.
Currently, the ability to abort an NVME LS request is limited to the nvme (host) side of the driver. In preparation of both the nvme and nvmet sides supporting Send LS Abort, rework the existing ls_req abort routines such that there is common code that can be used by both sides.
While refactoring it was seen the logic in the abort routine was incorrect. It attempted to abort all NVME LS's on the indicated port. As such, the routine was reworked to abort only the NVME LS request that was specified.
--- drivers/scsi/lpfc/lpfc_nvme.c | 125 +++++++++++++++++++++++++----------------- drivers/scsi/lpfc/lpfc_nvme.h | 2 + 2 files changed, 77 insertions(+), 50 deletions(-)
drivers/scsi/lpfc/lpfc_nvme.c +75 -50
@@ -790,83 +790,108 @@ lpfc_nvme_ls_req(struct nvme_fc_local_port *pnvme_lport, }/** -* lpfc_nvme_ls_abort - Issue an Link Service request -* @lpfc_pnvme: Pointer to the driver's nvme instance data -* @lpfc_nvme_lport: Pointer to the driver's local port data -* @lpfc_nvme_rport: Pointer to the rport getting the @lpfc_nvme_ereq +* __lpfc_nvme_ls_abort - Generic service routine to abort a prior +* NVME LS request +* @vport: The local port that issued the LS +* @ndlp: The remote port the LS was sent to +* @pnvme_lsreq: Pointer to LS request structure from the transport * -* Driver registers this routine to handle any link service request -* from the nvme_fc transport to a remote nvme-aware port. +* The driver validates the ndlp, looks for the LS, and aborts the +* LS if found. * -* Return value : -* 0 - Success -* TODO: What are the failure codes. +* Returns: +* 0 : if LS found and aborted +* non-zero: various error conditions in form -Exxx **/ -static void -lpfc_nvme_ls_abort (struct nvme_fc_local_port *pnvme_lport, -struct nvme_fc_remote_port *pnvme_rport, -struct nvmefc_ls_req *pnvme_lsreq) +int +__lpfc_nvme_ls_abort (struct lpfc_vport *vport,struct lpfc_nodelist *ndlp, +struct nvmefc_ls_req *pnvme_lsreq) { -struct lpfc_nvme_lport *lport; -struct lpfc_vport *vport; -struct lpfc_hba *phba; -struct lpfc_nodelist *ndlp; -LIST_HEAD (abort_list); +struct lpfc_hba *phba = vport->phba;struct lpfc_sli_ring *pring;struct lpfc_iocbq *wqe, *next_wqe; +bool foundit = false; - lport = (struct lpfc_nvme_lport *)pnvme_lport->private; -if (unlikely (!lport)) -return ; - vport = lport->vport; - phba = vport->phba; - -if (vport->load_flag &FC_UNLOADING ) -return ; - - ndlp =lpfc_findnode_did (vport, pnvme_rport->port_id);if (!ndlp) { -lpfc_printf_vlog (vport,KERN_ERR ,LOG_NVME_ABTS , -"6049 Could not find node for DID %x\n" , - pnvme_rport->port_id); -return ; +lpfc_printf_log (phba,KERN_ERR , +LOG_NVME_DISC |LOG_NODE | +LOG_NVME_IOERR |LOG_NVME_ABTS , +"6049 NVMEx LS REQ Abort: Bad NDLP x%px DID " +"x%06x, Failing LS Req\n" , + ndlp, ndlp ? ndlp->nlp_DID :0 ); +return -EINVAL ; } -/* Expand print to include key fields. */ -lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_ABTS , -"6040 ENTER. lport x%px, rport x%px lsreq x%px rqstlen:%d " -"rsplen:%d %pad %pad\n" , - pnvme_lport, pnvme_rport, +lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_DISC |LOG_NVME_ABTS , +"6040 NVMEx LS REQ Abort: Issue LS_ABORT for lsreq " +"x%p rqstlen:%d rsplen:%d %pad %pad\n" , pnvme_lsreq, pnvme_lsreq->rqstlen, pnvme_lsreq->rsplen, &pnvme_lsreq->rqstdma, &pnvme_lsreq->rspdma);/* -* Lock the ELS ring txcmplq and build a local list of all ELS IOs -* that need an ABTS. The IOs need to stay on the txcmplq so that -* the abort operation completes them successfully. +* Lock the ELS ring txcmplq and look for the wqe that matches +* this ELS. If found, issue an abort on the wqe. */ pring = phba->sli4_hba.nvmels_wq->pring;spin_lock_irq (&phba->hbalock);spin_lock (&pring->ring_lock);list_for_each_entry_safe (wqe, next_wqe, &pring->txcmplq, list) { -/* Add to abort_list on on NDLP match. */ -if (lpfc_check_sli_ndlp (phba, pring, wqe, ndlp)) { +if (wqe->context2 == pnvme_lsreq) { wqe->iocb_flag |=LPFC_DRIVER_ABORTED ; -list_add_tail (&wqe->dlist, &abort_list); + foundit = true; +break ; } }spin_unlock (&pring->ring_lock); + +if (foundit) +lpfc_sli_issue_abort_iotag (phba, pring, wqe);spin_unlock_irq (&phba->hbalock); -/* Abort the targeted IOs and remove them from the abort list. */ -list_for_each_entry_safe (wqe, next_wqe, &abort_list, dlist) { +if (foundit) +return 0 ; + +lpfc_printf_vlog (vport,KERN_INFO ,LOG_NVME_DISC |LOG_NVME_ABTS , +"6213 NVMEx LS REQ Abort: Unable to locate req x%p\n" , + pnvme_lsreq); +return 1 ; +} + +/** +* lpfc_nvme_ls_abort - Abort a prior NVME LS request +* @lpfc_nvme_lport: Transport localport that LS is to be issued from. +* @lpfc_nvme_rport: Transport remoteport that LS is to be sent to. +* @pnvme_lsreq - the transport nvme_ls_req structure for the LS +* +* Driver registers this routine to abort a NVME LS request that is +* in progress (from the transports perspective). +**/ +static void +lpfc_nvme_ls_abort (struct nvme_fc_local_port *pnvme_lport, +struct nvme_fc_remote_port *pnvme_rport, +struct nvmefc_ls_req *pnvme_lsreq) +{ +struct lpfc_nvme_lport *lport; +struct lpfc_vport *vport; +struct lpfc_hba *phba; +struct lpfc_nodelist *ndlp; +int ret; + + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; +if (unlikely (!lport)) +return ; + vport = lport->vport; + phba = vport->phba; + +if (vport->load_flag &FC_UNLOADING ) +return ; + + ndlp =lpfc_findnode_did (vport, pnvme_rport->port_id); + + ret =__lpfc_nvme_ls_abort (vport, ndlp, pnvme_lsreq); +if (!ret)atomic_inc (&lport->xmt_ls_abort); -spin_lock_irq (&phba->hbalock); -list_del_init (&wqe->dlist); -lpfc_sli_issue_abort_iotag (phba, pring, wqe); -spin_unlock_irq (&phba->hbalock); - } }/* Fix up the existing sgls for NVME IO. */
drivers/scsi/lpfc/lpfc_nvme.h +2 -0
@@ -238,6 +238,8 @@ int __lpfc_nvme_ls_req(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, struct lpfc_wcqe_complete *wcqe));void __lpfc_nvme_ls_req_cmp (struct lpfc_hba *phba,struct lpfc_vport *vport,struct lpfc_iocbq *cmdwqe,struct lpfc_wcqe_complete *wcqe); +int __lpfc_nvme_ls_abort (struct lpfc_vport *vport, +struct lpfc_nodelist *ndlp,struct nvmefc_ls_req *pnvme_lsreq);/* routines found in lpfc_nvmet.c */ int lpfc_nvme_unsol_ls_issue_abort(struct lpfc_hba *phba,
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 25/29] lpfc: Refactor Send LS Abort support
Quoted 11 lines · James Smart
Send LS Abort support is needed when Send LS Request is supported.
Currently, the ability to abort an NVME LS request is limited to the nvme (host) side of the driver. In preparation of both the nvme and nvmet sides supporting Send LS Abort, rework the existing
ls_reqabort routines such that there is common code that can be used by both sides.While refactoring it was seen the logic in the abort routine was incorrect. It attempted to abort all NVME LS's on the indicated port. As such, the routine was reworked to abort only the NVME LS request that was specified.
Quoted 7 lines
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvme.c | 125 +++++++++++++++++++++++++----------------- drivers/scsi/lpfc/lpfc_nvme.h | 2 + 2 files changed, 77 insertions(+), 50 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 26/29 lpfc: Refactor Send LS Response support
In reply to James SmartCurrently, the ability to send an NVME LS response is limited to the nvmet (controller/target) side of the driver. In preparation of both the nvme and nvmet sides supporting Send LS Response, rework the existing send ls_rsp and ls_rsp completion routines such that there is common code that can be used by both sides.
--- drivers/scsi/lpfc/lpfc_nvme.h | 7 ++ drivers/scsi/lpfc/lpfc_nvmet.c | 255 ++++++++++++++++++++++++++++------------- 2 files changed, 184 insertions(+), 78 deletions(-)
drivers/scsi/lpfc/lpfc_nvme.h +7 -0
@@ -245,3 +245,10 @@ int __lpfc_nvme_ls_abort(struct lpfc_vport *vport,int lpfc_nvme_unsol_ls_issue_abort (struct lpfc_hba *phba,struct lpfc_async_xchg_ctx *ctxp,uint32_t sid,uint16_t xri); +int __lpfc_nvme_xmt_ls_rsp (struct lpfc_async_xchg_ctx *axchg, +struct nvmefc_ls_rsp *ls_rsp, +void (*xmt_ls_rsp_cmp)(struct lpfc_hba *phba, +struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe)); +void __lpfc_nvme_xmt_ls_rsp_cmp (struct lpfc_hba *phba, +struct lpfc_iocbq *cmdwqe,struct lpfc_wcqe_complete *wcqe);
drivers/scsi/lpfc/lpfc_nvmet.c +177 -78
328 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 26/29] lpfc: Refactor Send LS Response support
Quoted 6 lines · James Smart
Currently, the ability to send an NVME LS response is limited to the nvmet (controller/target) side of the driver. In preparation of both the nvme and nvmet sides supporting Send LS Response, rework the existing send
ls_rspandls_rspcompletion routines such that there is common code that can be used by both sides.
Quoted 7 lines
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvme.h | 7 ++ drivers/scsi/lpfc/lpfc_nvmet.c | 255 ++++++++++++++++++++++++++++------------- 2 files changed, 184 insertions(+), 78 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 27/29 lpfc: nvme: Add Receive LS Request and Send LS Response support to nvme
In reply to James SmartNow that common helpers exist, add the ability to receive NVME LS requests to the driver. New requests will be delivered to the transport by nvme_fc_rcv_ls_req().
In order to complete the LS, add support for Send LS Response and send LS response completion handling to the driver.
--- drivers/scsi/lpfc/lpfc_nvme.c | 130 ++++++++++++++++++++++++++++++++++++++++++ drivers/scsi/lpfc/lpfc_nvme.h | 9 +++ 2 files changed, 139 insertions(+)
drivers/scsi/lpfc/lpfc_nvme.c +130 -0
@@ -400,6 +400,10 @@ lpfc_nvme_remoteport_delete(struct nvme_fc_remote_port *remoteport)* request. Any remaining validation is done and the LS is then forwarded * to the nvme-fc transport via nvme_fc_rcv_ls_req(). * +* The calling sequence should be: nvme_fc_rcv_ls_req() -> (processing) +* -> lpfc_nvme_xmt_ls_rsp/cmp -> req->done. +* lpfc_nvme_xmt_ls_rsp_cmp should free the allocated axchg. +* * Returns 0 if LS was handled and delivered to the transport * Returns 1 if LS failed to be handled and should be dropped */
@@ -407,6 +411,46 @@ int lpfc_nvme_handle_lsreq(struct lpfc_hba *phba, struct lpfc_async_xchg_ctx *axchg) { +#if (IS_ENABLED(CONFIG_NVME_FC)) + struct lpfc_vport *vport; +struct lpfc_nvme_rport *lpfc_rport; +struct nvme_fc_remote_port *remoteport; +struct lpfc_nvme_lport *lport; +uint32_t *payload = axchg->payload; +int rc; + + vport = axchg->ndlp->vport; + lpfc_rport = axchg->ndlp->nrport; +if (!lpfc_rport) +return -EINVAL ; + + remoteport = lpfc_rport->remoteport; +if (!vport->localport) +return -EINVAL ; + + lport = vport->localport->private; +if (!lport) +return -EINVAL ; + +atomic_inc (&lport->rcv_ls_req_in); + + rc =nvme_fc_rcv_ls_req (remoteport, &axchg->ls_rsp, axchg->payload, + axchg->size); + +lpfc_printf_log (phba,KERN_INFO ,LOG_NVME_DISC , +"6205 NVME Unsol rcv: sz %d rc %d: %08x %08x %08x " +"%08x %08x %08x\n" , + axchg->size, rc, + *payload, *(payload+1 ), *(payload+2 ), + *(payload+3 ), *(payload+4 ), *(payload+5 )); + +if (!rc) { +atomic_inc (&lport->rcv_ls_req_out); +return 0 ; + } + +atomic_inc (&lport->rcv_ls_req_drop); +#endif return 1 ; }
@@ -859,6 +903,81 @@ __lpfc_nvme_ls_abort(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, }/** +* lpfc_nvme_xmt_ls_rsp_cmp - Completion handler for LS Response +* @phba: Pointer to HBA context object. +* @cmdwqe: Pointer to driver command WQE object. +* @wcqe: Pointer to driver response CQE object. +* +* The function is called from SLI ring event handler with no +* lock held. This function is the completion handler for NVME LS commands +* The function updates any states and statistics, then calls the +* generic completion handler to free resources. +**/ +static void +lpfc_nvme_xmt_ls_rsp_cmp (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe) +{ +struct lpfc_vport *vport = cmdwqe->vport; +struct lpfc_nvme_lport *lport; +uint32_t status, result; + + status =bf_get (lpfc_wcqe_c_status, wcqe) &LPFC_IOCB_STATUS_MASK ; + result = wcqe->parameter; + +if (!vport->localport) + goto finish; + + lport = (struct lpfc_nvme_lport *)vport->localport->private; +if (lport) { +if (status) { +atomic_inc (&lport->xmt_ls_rsp_error); +if (result ==IOERR_ABORT_REQUESTED ) +atomic_inc (&lport->xmt_ls_rsp_aborted); +if (bf_get (lpfc_wcqe_c_xb, wcqe)) +atomic_inc (&lport->xmt_ls_rsp_xb_set); + }else { +atomic_inc (&lport->xmt_ls_rsp_cmpl); + } + } + +finish: +__lpfc_nvme_xmt_ls_rsp_cmp (phba, cmdwqe, wcqe); +} + +static int +lpfc_nvme_xmt_ls_rsp (struct nvme_fc_local_port *localport, +struct nvme_fc_remote_port *remoteport, +struct nvmefc_ls_rsp *ls_rsp) +{ +struct lpfc_async_xchg_ctx *axchg = + container_of(ls_rsp, struct lpfc_async_xchg_ctx, ls_rsp); +struct lpfc_nvme_lport *lport; +int rc; + +if (axchg->phba->pport->load_flag &FC_UNLOADING ) +return -ENODEV ; + + lport = (struct lpfc_nvme_lport *)localport->private; + + rc =__lpfc_nvme_xmt_ls_rsp (axchg, ls_rsp, lpfc_nvme_xmt_ls_rsp_cmp); + +if (rc) { +atomic_inc (&lport->xmt_ls_drop); +/* +* unless the failure is due to having already sent +* the response, an abort will be generated for the +* exchange if the rsp can't be sent. +*/ +if (rc != -EALREADY ) +atomic_inc (&lport->xmt_ls_abort); +return rc; + } + +atomic_inc (&lport->xmt_ls_rsp); +return 0 ; +} + +/** * lpfc_nvme_ls_abort - Abort a prior NVME LS request * @lpfc_nvme_lport: Transport localport that LS is to be issued from. * @lpfc_nvme_rport: Transport remoteport that LS is to be sent to.
@@ -2090,6 +2209,7 @@ static struct nvme_fc_port_template lpfc_nvme_template = { .fcp_io = lpfc_nvme_fcp_io_submit, .ls_abort = lpfc_nvme_ls_abort, .fcp_abort = lpfc_nvme_fcp_abort, + .xmt_ls_rsp = lpfc_nvme_xmt_ls_rsp, .max_hw_queues = 1, .max_sgl_segments = LPFC_NVME_DEFAULT_SEGS,
@@ -2285,6 +2405,16 @@ lpfc_nvme_create_localport(struct lpfc_vport *vport)atomic_set (&lport->cmpl_fcp_err,0 );atomic_set (&lport->cmpl_ls_xb,0 );atomic_set (&lport->cmpl_ls_err,0 ); +atomic_set (&lport->xmt_ls_rsp,0 ); +atomic_set (&lport->xmt_ls_drop,0 ); +atomic_set (&lport->xmt_ls_rsp_cmpl,0 ); +atomic_set (&lport->xmt_ls_rsp_error,0 ); +atomic_set (&lport->xmt_ls_rsp_aborted,0 ); +atomic_set (&lport->xmt_ls_rsp_xb_set,0 ); +atomic_set (&lport->rcv_ls_req_in,0 ); +atomic_set (&lport->rcv_ls_req_out,0 ); +atomic_set (&lport->rcv_ls_req_drop,0 ); +atomic_set (&lport->fc4NvmeLsRequests,0 );atomic_set (&lport->fc4NvmeLsCmpls,0 ); }
drivers/scsi/lpfc/lpfc_nvme.h +9 -0
@@ -67,6 +67,15 @@ struct lpfc_nvme_lport {atomic_t cmpl_fcp_err;atomic_t cmpl_ls_xb;atomic_t cmpl_ls_err; +atomic_t xmt_ls_rsp; +atomic_t xmt_ls_drop; +atomic_t xmt_ls_rsp_cmpl; +atomic_t xmt_ls_rsp_error; +atomic_t xmt_ls_rsp_aborted; +atomic_t xmt_ls_rsp_xb_set; +atomic_t rcv_ls_req_in; +atomic_t rcv_ls_req_out; +atomic_t rcv_ls_req_drop; };struct lpfc_nvme_rport {
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 27/29] lpfc: nvme: Add Receive LS Request and Send LS Response support to nvme
Quoted 194 lines · James Smart
Now that common helpers exist, add the ability to receive NVME LS requests to the driver. New requests will be delivered to the transport by
nvme_fc_rcv_ls_req().In order to complete the LS, add support for Send LS Response and send LS response completion handling to the driver.
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvme.c | 130 ++++++++++++++++++++++++++++++++++++++++++ drivers/scsi/lpfc/lpfc_nvme.h | 9 +++ 2 files changed, 139 insertions(+)drivers/scsi/lpfc/lpfc_nvme.c +130
-0@@ -400,6 +400,10 @@ lpfc_nvme_remoteport_delete(struct nvme_fc_remote_port *remoteport)* request. Any remaining validation is done and the LS is then forwarded * to the nvme-fc transport via nvme_fc_rcv_ls_req(). * +* The calling sequence should be: nvme_fc_rcv_ls_req() -> (processing) +* -> lpfc_nvme_xmt_ls_rsp/cmp -> req->done. +* lpfc_nvme_xmt_ls_rsp_cmp should free the allocated axchg. +* * Returns 0 if LS was handled and delivered to the transport * Returns 1 if LS failed to be handled and should be dropped */ @@ -407,6 +411,46 @@ int lpfc_nvme_handle_lsreq(struct lpfc_hba *phba, struct lpfc_async_xchg_ctx *axchg) { +#if (IS_ENABLED(CONFIG_NVME_FC)) + struct lpfc_vport *vport; +struct lpfc_nvme_rport *lpfc_rport; +struct nvme_fc_remote_port *remoteport; +struct lpfc_nvme_lport *lport; +uint32_t *payload = axchg->payload; +int rc; + + vport = axchg->ndlp->vport; + lpfc_rport = axchg->ndlp->nrport; +if (!lpfc_rport) +return -EINVAL ; + + remoteport = lpfc_rport->remoteport; +if (!vport->localport) +return -EINVAL ; + + lport = vport->localport->private; +if (!lport) +return -EINVAL ; + +atomic_inc (&lport->rcv_ls_req_in); + + rc =nvme_fc_rcv_ls_req (remoteport, &axchg->ls_rsp, axchg->payload, + axchg->size); + +lpfc_printf_log (phba,KERN_INFO ,LOG_NVME_DISC , +"6205 NVME Unsol rcv: sz %d rc %d: %08x %08x %08x " +"%08x %08x %08x\n" , + axchg->size, rc, + *payload, *(payload+1 ), *(payload+2 ), + *(payload+3 ), *(payload+4 ), *(payload+5 )); + +if (!rc) { +atomic_inc (&lport->rcv_ls_req_out); +return 0 ; + } + +atomic_inc (&lport->rcv_ls_req_drop); +#endif return 1 ; }@@ -859,6 +903,81 @@ __lpfc_nvme_ls_abort(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, }/** +* lpfc_nvme_xmt_ls_rsp_cmp - Completion handler for LS Response +* @phba: Pointer to HBA context object. +* @cmdwqe: Pointer to driver command WQE object. +* @wcqe: Pointer to driver response CQE object. +* +* The function is called from SLI ring event handler with no +* lock held. This function is the completion handler for NVME LS commands +* The function updates any states and statistics, then calls the +* generic completion handler to free resources. +**/ +static void +lpfc_nvme_xmt_ls_rsp_cmp (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe) +{ +struct lpfc_vport *vport = cmdwqe->vport; +struct lpfc_nvme_lport *lport; +uint32_t status, result; + + status =bf_get (lpfc_wcqe_c_status, wcqe) &LPFC_IOCB_STATUS_MASK ; + result = wcqe->parameter; + +if (!vport->localport) + goto finish; + + lport = (struct lpfc_nvme_lport *)vport->localport->private; +if (lport) { +if (status) { +atomic_inc (&lport->xmt_ls_rsp_error); +if (result ==IOERR_ABORT_REQUESTED ) +atomic_inc (&lport->xmt_ls_rsp_aborted); +if (bf_get (lpfc_wcqe_c_xb, wcqe)) +atomic_inc (&lport->xmt_ls_rsp_xb_set); + }else { +atomic_inc (&lport->xmt_ls_rsp_cmpl); + } + } + +finish: +__lpfc_nvme_xmt_ls_rsp_cmp (phba, cmdwqe, wcqe); +} + +static int +lpfc_nvme_xmt_ls_rsp (struct nvme_fc_local_port *localport, +struct nvme_fc_remote_port *remoteport, +struct nvmefc_ls_rsp *ls_rsp) +{ +struct lpfc_async_xchg_ctx *axchg = + container_of(ls_rsp, struct lpfc_async_xchg_ctx, ls_rsp); +struct lpfc_nvme_lport *lport; +int rc; + +if (axchg->phba->pport->load_flag &FC_UNLOADING ) +return -ENODEV ; + + lport = (struct lpfc_nvme_lport *)localport->private; + + rc =__lpfc_nvme_xmt_ls_rsp (axchg, ls_rsp, lpfc_nvme_xmt_ls_rsp_cmp); + +if (rc) { +atomic_inc (&lport->xmt_ls_drop); +/* +* unless the failure is due to having already sent +* the response, an abort will be generated for the +* exchange if the rsp can't be sent. +*/ +if (rc != -EALREADY ) +atomic_inc (&lport->xmt_ls_abort); +return rc; + } + +atomic_inc (&lport->xmt_ls_rsp); +return 0 ; +} + +/** * lpfc_nvme_ls_abort - Abort a prior NVME LS request * @lpfc_nvme_lport: Transport localport that LS is to be issued from. * @lpfc_nvme_rport: Transport remoteport that LS is to be sent to. @@ -2090,6 +2209,7 @@ static struct nvme_fc_port_template lpfc_nvme_template = { .fcp_io = lpfc_nvme_fcp_io_submit, .ls_abort = lpfc_nvme_ls_abort, .fcp_abort = lpfc_nvme_fcp_abort, + .xmt_ls_rsp = lpfc_nvme_xmt_ls_rsp, .max_hw_queues = 1, .max_sgl_segments = LPFC_NVME_DEFAULT_SEGS,@@ -2285,6 +2405,16 @@ lpfc_nvme_create_localport(struct lpfc_vport *vport)atomic_set (&lport->cmpl_fcp_err,0 );atomic_set (&lport->cmpl_ls_xb,0 );atomic_set (&lport->cmpl_ls_err,0 ); +atomic_set (&lport->xmt_ls_rsp,0 ); +atomic_set (&lport->xmt_ls_drop,0 ); +atomic_set (&lport->xmt_ls_rsp_cmpl,0 ); +atomic_set (&lport->xmt_ls_rsp_error,0 ); +atomic_set (&lport->xmt_ls_rsp_aborted,0 ); +atomic_set (&lport->xmt_ls_rsp_xb_set,0 ); +atomic_set (&lport->rcv_ls_req_in,0 ); +atomic_set (&lport->rcv_ls_req_out,0 ); +atomic_set (&lport->rcv_ls_req_drop,0 ); +atomic_set (&lport->fc4NvmeLsRequests,0 );atomic_set (&lport->fc4NvmeLsCmpls,0 ); }drivers/scsi/lpfc/lpfc_nvme.h +2
-0@@ -67,6 +67,15 @@ struct lpfc_nvme_lport {atomic_t cmpl_fcp_err;atomic_t cmpl_ls_xb;atomic_t cmpl_ls_err; +atomic_t xmt_ls_rsp; +atomic_t xmt_ls_drop;
drivers/scsi/lpfc/lpfc_nvme.h struct lpfc_nvme_lport { · line 81
Quoted 10 lines
drivers/scsi/lpfc/lpfc_nvme.h +7
-0+atomic_t xmt_ls_rsp_cmpl; +atomic_t xmt_ls_rsp_error; +atomic_t xmt_ls_rsp_aborted; +atomic_t xmt_ls_rsp_xb_set; +atomic_t rcv_ls_req_in; +atomic_t rcv_ls_req_out; +atomic_t rcv_ls_req_drop; };struct lpfc_nvme_rport {
And here's me, worrying about the impact a single atomic_t might have ... Any ideas of making this conditional and allowing to enable statistics separately?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 28/29 lpfc: nvmet: Add support for NVME LS request hosthandle
In reply to James SmartAs the nvmet layer does not have the concept of a remoteport object, which can be used to identify the entity on the other end of the fabric that is to receive an LS, the hosthandle was introduced. The driver passes the hosthandle, a value representative of the remote port, with a ls request receive. The LS request will create the association. The transport will remember the hosthandle for the association, and if there is a need to initiate a LS request to the remote port for the association, the hosthandle will be used. When the driver loses connectivity with the remote port, it needs to notify the transport that the hosthandle is no longer valid, allowing the transport to terminate associations related to the hosthandle.
This patch adds support to the driver for the hosthandle. The driver will use the ndlp pointer of the remote port for the hosthandle in calls to nvmet_fc_rcv_ls_req(). The discovery engine is updated to invalidate the hosthandle whenever connectivity with the remote port is lost.
--- drivers/scsi/lpfc/lpfc_crtn.h | 2 ++ drivers/scsi/lpfc/lpfc_hbadisc.c | 6 +++++ drivers/scsi/lpfc/lpfc_nportdisc.c | 11 ++++++++ drivers/scsi/lpfc/lpfc_nvme.h | 3 +++ drivers/scsi/lpfc/lpfc_nvmet.c | 53 +++++++++++++++++++++++++++++++++++++- 5 files changed, 74 insertions(+), 1 deletion(-)
drivers/scsi/lpfc/lpfc_crtn.h +2 -0
@@ -572,6 +572,8 @@ void lpfc_nvmet_unsol_fcp_event(struct lpfc_hba *phba, uint32_t idx, struct rqb_dmabuf *nvmebuf, uint64_t isr_ts, uint8_t cqflag);void lpfc_nvme_mod_param_dep (struct lpfc_hba *phba); +void lpfc_nvmet_invalidate_host (struct lpfc_hba *phba, +struct lpfc_nodelist *ndlp);void lpfc_nvme_abort_fcreq_cmpl (struct lpfc_hba *phba,struct lpfc_iocbq *cmdiocb,struct lpfc_wcqe_complete *abts_cmpl);
drivers/scsi/lpfc/lpfc_hbadisc.c +6 -0
@@ -822,6 +822,12 @@ lpfc_cleanup_rpis(struct lpfc_vport *vport, int remove)if ((phba->sli_rev <LPFC_SLI_REV4 ) && (!remove && ndlp->nlp_type &NLP_FABRIC ))continue ; + +/* Notify transport of connectivity loss to trigger cleanup. */ +if (phba->nvmet_support && + ndlp->nlp_state ==NLP_STE_UNMAPPED_NODE ) +lpfc_nvmet_invalidate_host (phba, ndlp); + lpfc_disc_state_machine(vport, ndlp, NULL, remove ? NLP_EVT_DEVICE_RM
drivers/scsi/lpfc/lpfc_nportdisc.c +11 -0
@@ -434,6 +434,11 @@ lpfc_rcv_plogi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, (unsigned long long ) wwn_to_u64(sp->portName.u.wwn)); +/* Notify transport of connectivity loss to trigger cleanup. */ +if (phba->nvmet_support && + ndlp->nlp_state ==NLP_STE_UNMAPPED_NODE ) +lpfc_nvmet_invalidate_host (phba, ndlp); + ndlp->nlp_prev_state = ndlp->nlp_state;/* rport needs to be unregistered first */ lpfc_nlp_set_state (vport, ndlp,NLP_STE_NPR_NODE );
@@ -749,6 +754,12 @@ lpfc_rcv_logo(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp,lpfc_els_rsp_acc (vport,ELS_CMD_PRLO , cmdiocb, ndlp,NULL ); elselpfc_els_rsp_acc (vport ,ELS_CMD_ACC ,cmdiocb ,ndlp ,NULL ); + +/* Notify transport of connectivity loss to trigger cleanup. */ +if (phba->nvmet_support && + ndlp->nlp_state ==NLP_STE_UNMAPPED_NODE ) +lpfc_nvmet_invalidate_host (phba, ndlp); +if (ndlp->nlp_DID == Fabric_DID) {if (vport->port_state <=LPFC_FDISC ) goto out;
drivers/scsi/lpfc/lpfc_nvme.h +3 -0
@@ -108,9 +108,12 @@ struct lpfc_nvme_fcpreq_priv {#define LPFC_NVMET_WAIT_TMO (5 * MSEC_PER_SEC)/* Used for NVME Target */ +#define LPFC_NVMET_INV_HOST_ACTIVE 1 +struct lpfc_nvmet_tgtport {struct lpfc_hba *phba;struct completion *tport_unreg_cmp; +atomic_t state;/* tracks nvmet hosthandle invalidation */ /* Stats counters - lpfc_nvmet_unsol_ls_buffer */ atomic_t rcv_ls_req_in;
drivers/scsi/lpfc/lpfc_nvmet.c +52 -1
74 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 28/29] lpfc: nvmet: Add support for NVME LS request hosthandle
Quoted 19 lines · James Smart
As the nvmet layer does not have the concept of a remoteport object, which can be used to identify the entity on the other end of the fabric that is to receive an LS, the hosthandle was introduced. The driver passes the hosthandle, a value representative of the remote port, with a ls request receive. The LS request will create the association. The transport will remember the hosthandle for the association, and if there is a need to initiate a LS request to the remote port for the association, the hosthandle will be used. When the driver loses connectivity with the remote port, it needs to notify the transport that the hosthandle is no longer valid, allowing the transport to terminate associations related to the hosthandle.
This patch adds support to the driver for the hosthandle. The driver will use the ndlp pointer of the remote port for the hosthandle in calls to
nvmet_fc_rcv_ls_req(). The discovery engine is updated to invalidate the hosthandle whenever connectivity with the remote port is lost.Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>
Quoted 8 lines
--- drivers/scsi/lpfc/lpfc_crtn.h | 2 ++ drivers/scsi/lpfc/lpfc_hbadisc.c | 6 +++++ drivers/scsi/lpfc/lpfc_nportdisc.c | 11 ++++++++ drivers/scsi/lpfc/lpfc_nvme.h | 3 +++ drivers/scsi/lpfc/lpfc_nvmet.c | 53 +++++++++++++++++++++++++++++++++++++- 5 files changed, 74 insertions(+), 1 deletion(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 29/29 lpfc: nvmet: Add Send LS Request and Abort LS Request support
In reply to James SmartNow that common helpers exist, add the ability to Send an NVME LS Request and to Abort an outstanding LS Request to the nvmet side of the driver.
--- drivers/scsi/lpfc/lpfc_nvme.h | 8 +++ drivers/scsi/lpfc/lpfc_nvmet.c | 128 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+)
drivers/scsi/lpfc/lpfc_nvme.h +8 -0
@@ -166,6 +166,14 @@ struct lpfc_nvmet_tgtport {atomic_t defer_ctx;atomic_t defer_fod;atomic_t defer_wqfull; + +/* Stats counters - ls_reqs, ls_aborts, host_invalidate */ +atomic_t xmt_ls_reqs; +atomic_t xmt_ls_cmpls; +atomic_t xmt_ls_err; +atomic_t cmpl_ls_err; +atomic_t cmpl_ls_xb; +atomic_t cmpl_ls_reqs; };struct lpfc_nvmet_ctx_info {
drivers/scsi/lpfc/lpfc_nvmet.c +128 -0
158 lines not on this page · raw message
Signature · 7 lines
2.13.7 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 29/29] lpfc: nvmet: Add Send LS Request and Abort LS Request support
Quoted 187 lines · James Smart
Now that common helpers exist, add the ability to Send an NVME LS Request and to Abort an outstanding LS Request to the nvmet side of the driver.
Signed-off-by Paul Ely <[email protected]>Signed-off-by James Smart <[email protected]>--- drivers/scsi/lpfc/lpfc_nvme.h | 8 +++ drivers/scsi/lpfc/lpfc_nvmet.c | 128 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+)drivers/scsi/lpfc/lpfc_nvme.h +8
-0@@ -166,6 +166,14 @@ struct lpfc_nvmet_tgtport {atomic_t defer_ctx;atomic_t defer_fod;atomic_t defer_wqfull; + +/* Stats counters - ls_reqs, ls_aborts, host_invalidate */ +atomic_t xmt_ls_reqs; +atomic_t xmt_ls_cmpls; +atomic_t xmt_ls_err; +atomic_t cmpl_ls_err; +atomic_t cmpl_ls_xb; +atomic_t cmpl_ls_reqs; };struct lpfc_nvmet_ctx_info {drivers/scsi/lpfc/lpfc_nvmet.c +122
-0@@ -1283,6 +1283,122 @@ lpfc_nvmet_defer_rcv(struct nvmet_fc_target_port *tgtport,spin_unlock_irqrestore (&ctxp->ctxlock, iflag); } +/** +* lpfc_nvmet_ls_req_cmp - completion handler for a nvme ls request +* @phba: Pointer to HBA context object +* @cmdwqe: Pointer to driver command WQE object. +* @wcqe: Pointer to driver response CQE object. +* +* This function is the completion handler for NVME LS requests. +* The function updates any states and statistics, then calls the +* generic completion handler to finish completion of the request. +**/ +static void +lpfc_nvmet_ls_req_cmp (struct lpfc_hba *phba,struct lpfc_iocbq *cmdwqe, +struct lpfc_wcqe_complete *wcqe) +{ +struct lpfc_vport *vport = cmdwqe->vport; +uint32_t status; +struct lpfc_nvmet_tgtport *tgtp; + + status =bf_get (lpfc_wcqe_c_status, wcqe) &LPFC_IOCB_STATUS_MASK ; + +if (!phba->targetport) + goto finish; + + tgtp = phba->targetport->private; +if (tgtp) { +atomic_inc (&tgtp->cmpl_ls_reqs); +if (status) { +if (bf_get (lpfc_wcqe_c_xb, wcqe)) +atomic_inc (&tgtp->cmpl_ls_xb); +atomic_inc (&tgtp->cmpl_ls_err); + } + } + +finish: +__lpfc_nvme_ls_req_cmp (phba, vport, cmdwqe, wcqe); +} + +/** +* lpfc_nvmet_ls_req - Issue an Link Service request +* @targetport - pointer to target instance registered with nvmet transport. +* @hosthandle - hosthandle set by the driver in a prior ls_rqst_rcv. +* Driver sets this value to the ndlp pointer. +* @pnvme_lsreq - the transport nvme_ls_req structure for the LS +* +* Driver registers this routine to handle any link service request +* from the nvme_fc transport to a remote nvme-aware port. +* +* Return value : +* 0 - Success +* non-zero: various error codes, in form of -Exxx +**/ +static int +lpfc_nvmet_ls_req (struct nvmet_fc_target_port *targetport, +void *hosthandle, +struct nvmefc_ls_req *pnvme_lsreq) +{ +struct lpfc_nvmet_tgtport *lpfc_nvmet = targetport->private; +struct lpfc_hba *phba; +struct lpfc_nodelist *ndlp; +int ret; +u32 hstate; + +if (!lpfc_nvmet) +return -EINVAL ; + + phba = lpfc_nvmet->phba; +if (phba->pport->load_flag &FC_UNLOADING ) +return -EINVAL ; + + hstate =atomic_read (&lpfc_nvmet->state); +if (hstate ==LPFC_NVMET_INV_HOST_ACTIVE ) +return -EACCES ; + + ndlp = (struct lpfc_nodelist *)hosthandle; + +atomic_inc (&lpfc_nvmet->xmt_ls_reqs); + + ret =__lpfc_nvme_ls_req (phba->pport, ndlp, pnvme_lsreq, + lpfc_nvmet_ls_req_cmp); +if (ret) +atomic_inc (&lpfc_nvmet->xmt_ls_err); + +return ret; +} + +/** +* lpfc_nvmet_ls_abort - Abort a prior NVME LS request +* @targetport: Transport targetport, that LS was issued from. +* @hosthandle - hosthandle set by the driver in a prior ls_rqst_rcv. +* Driver sets this value to the ndlp pointer. +* @pnvme_lsreq - the transport nvme_ls_req structure for LS to be aborted +* +* Driver registers this routine to abort an NVME LS request that is +* in progress (from the transports perspective). +**/ +static void +lpfc_nvmet_ls_abort (struct nvmet_fc_target_port *targetport, +void *hosthandle, +struct nvmefc_ls_req *pnvme_lsreq) +{ +struct lpfc_nvmet_tgtport *lpfc_nvmet = targetport->private; +struct lpfc_hba *phba; +struct lpfc_nodelist *ndlp; +int ret; + + phba = lpfc_nvmet->phba; +if (phba->pport->load_flag &FC_UNLOADING ) +return ; + + ndlp = (struct lpfc_nodelist *)hosthandle; + + ret =__lpfc_nvme_ls_abort (phba->pport, ndlp, pnvme_lsreq); +if (!ret) +atomic_inc (&lpfc_nvmet->xmt_ls_abort); +} +static void lpfc_nvmet_host_release (void *hosthandle) {@@ -1325,6 +1441,8 @@ static struct nvmet_fc_target_template lpfc_tgttemplate = { .fcp_req_release = lpfc_nvmet_xmt_fcp_release, .defer_rcv = lpfc_nvmet_defer_rcv, .discovery_event = lpfc_nvmet_discovery_event, + .ls_req = lpfc_nvmet_ls_req, + .ls_abort = lpfc_nvmet_ls_abort, .host_release = lpfc_nvmet_host_release, .max_hw_queues = 1,@@ -1336,6 +1454,7 @@ static struct nvmet_fc_target_template lpfc_tgttemplate = { .target_features = 0, /* sizes of additional private data for data structures */ .target_priv_sz = sizeof(struct lpfc_nvmet_tgtport), + .lsrqst_priv_sz = 0, }; static void@@ -1638,6 +1757,9 @@ lpfc_nvmet_create_targetport(struct lpfc_hba *phba)atomic_set (&tgtp->xmt_fcp_xri_abort_cqe,0 );atomic_set (&tgtp->xmt_fcp_abort,0 );atomic_set (&tgtp->xmt_fcp_abort_cmpl,0 ); +atomic_set (&tgtp->xmt_ls_reqs,0 ); +atomic_set (&tgtp->xmt_ls_cmpls,0 ); +atomic_set (&tgtp->xmt_ls_err,0 );atomic_set (&tgtp->xmt_abort_unsol,0 );atomic_set (&tgtp->xmt_abort_sol,0 );atomic_set (&tgtp->xmt_abort_rsp,0 );@@ -1645,6 +1767,12 @@ lpfc_nvmet_create_targetport(struct lpfc_hba *phba)atomic_set (&tgtp->defer_ctx,0 );atomic_set (&tgtp->defer_fod,0 );
drivers/scsi/lpfc/lpfc_nvmet.c lpfc_nvmet_create_targetport(struct lpfc_hba *phba) · line 1778
Quoted 10 lines
drivers/scsi/lpfc/lpfc_nvmet.c +6
-0atomic_set (&tgtp->defer_wqfull,0 ); +atomic_set (&tgtp->xmt_ls_reqs,0 ); +atomic_set (&tgtp->xmt_ls_cmpls,0 ); +atomic_set (&tgtp->xmt_ls_err,0 ); +atomic_set (&tgtp->cmpl_ls_err,0 ); +atomic_set (&tgtp->cmpl_ls_xb,0 ); +atomic_set (&tgtp->cmpl_ls_reqs,0 ); }return error; }
Same story: any ideas of making this conditional to allow for enabling statistics separately?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 116 lines · James Smart
At the tail end of FC-NVME (1) standardization, the process for terminating an association was changed requiring interlock using FC-NVME Disconnect Assocation LS's between both the host port and target port. This was immediately relaxed with an ammendment to FC-NVME (1) and with wording put into FC-NVME-2. The interlock was removed, but it still required both the host port and target port to initiate Disconnect Association LS's and respond to LS's.
The linux nvme-fc and nvmet-fc implementations were interoperable with standards but the linux-driver api did not support all the functionality needed. It was missing:
- nvme-fc: didn't support the reception of NVME LS's and the ability to transmit responses to an LS.
- nvmet-fc: didn't support the ability to send an NVME LS request. It also did not support a method for the transport to specify a remote port for an LS.This patch adds the missing functionality. Specifically the patch set:
- Updates the header with the FC-NVME-2 standard out for final approval.
- Refactors data structure names that used to be dependent on role (ls requests were specific to nvme; ls responses were specific to nvmet) to generic names that can be used by both nvme-fc and nvmet-fc.
- Modifies the nvme-fc transport template with interfaces to receive NVME LS's and for the transport to then request LS responses to be sent.
- Modifies the nvmet-fc transport template with:
- The current NVME LS receive interface was modified to supply a handle to indentify the remote port the LS as received from. If the LS creates an association, the handle may be used to initiate NVME LS requests to the remote port. An interface was put in place to invalidate the handle on connectivity losses.
- Interfaces for the transport to request an NVME LS request to be performed as well as to abort that LS in cases of error/teardown.
- The nvme-fc transport was modified to follow the standard:
- Disconnect association logic was revised to send Disconnect LS as soon as all ABTS's were transmit rather than waiting for the ABTS process to fully complete.
- Disconnect LS reception is supported, with reception initiating controller reset and reconnect.
- Disconnect LS responses will not be transmit until association termination has transmit the Disconnect LS.
- The nvmet-fc transport was modified to follow the standard:
- Disconnect assocation logic was revised to transmit a Disconnect LS request as soon as all ABTS's have been transmit. In the past, no Disconnect LS had been transmit.
- Disconnect LS responses will not be sent until the Disconnect LS request has been transmit.
- nvme-fcloop: was updated with interfaces to allow testing of the transports.
- Along the way, cleanups and slight corrections were made to the transports.
- The lpfc driver was modified to support the new transport interfaces for both the nvme and nvmet transports. As much of the functionality was already present, but specific to one side of the transport, existing code was refactored to create common routines. Addition of the new interfaces was able to slip in rather easily with the common routines.This code was cut against the for-5.6 branch.
I'll work with Martin to minimize any work to merge these lpfc mods with lpfc changes in the scsi tree.
-- james
James Smart (29):
nvme-fc: Sync header to FC-NVME-2 rev 1.08
nvmet-fc: fix typo in comment
nvme-fc and nvmet-fc: revise LLDD api for LS reception and LS request nvme-fcnvmet_fcnvme_fcloop: adapt code to changed names in api
header
lpfc: adapt code to changed names in api header
nvme-fcloop: Fix deallocation of working context
nvme-fc nvmet-fc: refactor for common LS definitions
nvmet-fc: Better size LS buffers
nvme-fc: Ensure private pointers are NULL if no data
nvmefc: Use common definitions for LS names, formatting, and
validation
nvme-fc: convertassoc_activeflag to atomic
nvme-fc: Add Disconnect Association Rcv support
nvmet-fc: add LS failure messages
nvmet-fc: perform small cleanups on unneeded checks
nvmet-fc: track hostport handle for associations
nvmet-fc: renamels_listtols_rcv_list
nvmet-fc: Add Disconnect Association Xmt support
nvme-fcloop: refactor to enable target to host LS
nvme-fcloop: add target to host LS request support
lpfc: Refactor lpfc nvme headers
lpfc: Refactornvmet_rcv_ctxto createlpfc_async_xchg_ctx
lpfc: Commonizelpfc_async_xchg_ctxstate and flag definitions
lpfc: Refactor NVME LS receive handling
lpfc: Refactor Send LS Request support
lpfc: Refactor Send LS Abort support
lpfc: Refactor Send LS Response support
lpfc: nvme: Add Receive LS Request and Send LS Response support to
nvme
lpfc: nvmet: Add support for NVME LS request hosthandle
lpfc: nvmet: Add Send LS Request and Abort LS Request supportdrivers/nvme/host/fc.c | 555 +++++++++++++++++++----- drivers/nvme/host/fc.h | 227 ++++++++++ drivers/nvme/target/fc.c | 800 +++++++++++++++++++++++++---------- drivers/nvme/target/fcloop.c | 228 ++++++++-- drivers/scsi/lpfc/lpfc.h | 2 +- drivers/scsi/lpfc/lpfc_attr.c | 3 - drivers/scsi/lpfc/lpfc_crtn.h | 9 +- drivers/scsi/lpfc/lpfc_ct.c | 1 - drivers/scsi/lpfc/lpfc_debugfs.c | 5 +- drivers/scsi/lpfc/lpfc_hbadisc.c | 8 +- drivers/scsi/lpfc/lpfc_init.c | 7 +- drivers/scsi/lpfc/lpfc_mem.c | 4 - drivers/scsi/lpfc/lpfc_nportdisc.c | 13 +- drivers/scsi/lpfc/lpfc_nvme.c | 550 ++++++++++++++++-------- drivers/scsi/lpfc/lpfc_nvme.h | 198 +++++++++
Quoted 9 lines
drivers/scsi/lpfc/lpfc_nvmet.c | 837 +++++++++++++++++++++++-------------- drivers/scsi/lpfc/lpfc_nvmet.h | 158 ------- drivers/scsi/lpfc/lpfc_sli.c | 126 +++++- include/linux/nvme-fc-driver.h | 368 +++++++++++----- include/linux/nvme-fc.h | 9 +- 20 files changed, 2970 insertions(+), 1138 deletions(-) create mode 100644 drivers/nvme/host/fc.h delete mode 100644 drivers/scsi/lpfc/lpfc_nvmet.h
As mentioned, this patchset resolves one crash I've found when running my nvme/034 blktest. However, we're still not quite there yet, as the very same test now results in:
[ 686.272596] (NULL device *): {1:0} Association deleted
[ 686.318243] (NULL device *): {1:0} Association freed
[ 1309.673673] kmemleak: 2 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)
[ 1488.024222] INFO: task kworker/4:2:656 blocked for more than 491 seconds.
[ 1488.085196] Tainted: G E 5.6.0-rc1-default+ #518
[ 1488.142347] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1488.211323] kworker/4:2 D 0 656 2 0x80004000
[ 1488.211329] Workqueue: events fcloop_tport_lsrqst_work [nvme_fcloop]
[ 1488.211331] Call Trace:
[ 1488.211337] ? __schedule+0x28b/0x720
[ 1488.369706] schedule+0x40/0xb0
[ 1488.369708] schedule_timeout+0x1dd/0x300
[ 1488.369713] ? free_pcp_prepare+0x59/0x1d0
[ 1488.469700] wait_for_completion+0xba/0x140
[ 1488.469702] ? wake_up_q+0xa0/0xa0
[ 1488.469705] __flush_work+0x177/0x1b0
[ 1488.469708] ? worker_detach_from_pool+0xa0/0xa0
[ 1488.610280] fcloop_targetport_delete+0x13/0x20 [nvme_fcloop]
[ 1488.610284] nvmet_fc_tgtport_put+0x150/0x190 [nvmet_fc]
[ 1488.706952] nvmet_fc_disconnect_assoc_done+0x9a/0xe0 [nvmet_fc]
[ 1488.706954] fcloop_tport_lsrqst_work+0x7a/0xa0 [nvme_fcloop]
[ 1488.706957] process_one_work+0x208/0x400
[ 1488.706959] worker_thread+0x2d/0x3e0
[ 1488.706961] ? process_one_work+0x400/0x400
[ 1488.706963] kthread+0x117/0x130
[ 1488.706966] ? kthread_create_worker_on_cpu+0x70/0x70
[ 1488.985004] ret_from_fork+0x35/0x40So we do miss a completion somewhere ...
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
James,
can you resend this with the nitpicks address and the reviewed-bys collected?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme