Add VMID support to nvme-fc transport and lpfc driver
Subsystems (4)
v2 cover Add VMID support to nvme-fc transport and lpfc driver
This patch adds vmid support to the nvme-fc transport.
Various virtualization technologies used in Fibre Channel SAN deployments have the ability to identify and associate traffic with specific virtualized applications. The T11 standard defines an application services tag that can be added to FC traffic to aid in identification and monitoring of traffic associated with the applications.
VMID support is present in the kernel in blkcg, the SCSI fc transport has tied into the infrastructure, and libvirt has been updated to support the blkcg settings. Refer to:
https://lore.kernel.org/all/[email protected]/
This patch set ties the nvme-fc transport into the blkcg infrastructure so that vmid tags can be added to nvme traffic. The patch set also updates the lpfc driver to utilize the nvme-fc transport addition.
Although the patch adds a nvme interface, it is being sent to the SCSI maintainers tree due to the lpfc dependencies. As there is no other consumer of the new interface, when the scsi tree merges with mainline, the new interface will be picked up there.
Patches cut against scsi 5.19/scsi-queue tree with lpfc 14.2.0.3 patches included.
V2:
Revised the first patch which calls blkcg_get_fc_appid(). Fixes build errors when merged with
blk-cgroup: move blkcg_{get,set}_fc_appid out of line commit db05628435aa
James Smart (3):
lpfc: commonize VMID code location
lpfc: rework lpfc_vmid_get_appid() to be protocol independent
lpfc: Add support for vmid tagging of NVMe I/Os
Muneendra (1):
nvme-fc: Add new routine nvme_fc_io_getuuid
drivers/nvme/host/fc.c | 18 +++ drivers/scsi/lpfc/Makefile | 2 +- drivers/scsi/lpfc/lpfc_crtn.h | 3 + drivers/scsi/lpfc/lpfc_nvme.c | 45 ++++++ drivers/scsi/lpfc/lpfc_scsi.c | 263 +----------------------------- drivers/scsi/lpfc/lpfc_vmid.c | 288 +++++++++++++++++++++++++++++++++ include/linux/nvme-fc-driver.h | 14 ++ 7 files changed, 373 insertions(+), 260 deletions(-) create mode 100644 drivers/scsi/lpfc/lpfc_vmid.c
Signature · 1 line
2.26.2
PATCH v2 1/4 nvme-fc: Add new routine nvme_fc_io_getuuid
From: Muneendra <[email protected]>
Add nvme_fc_io_getuuid() to the nvme-fc transport.
The routine is invoked by the fc LLDD on a per-io request basis. The routine translates from the fc-specific request structure to the bio and the cgroup structure in order to obtain the fc appid stored in the cgroup structure. If a value is not set or a bio is not found, a NULL appid (aka uuid) will be returned to the LLDD.
---
v2: modify to sync with Christoph's patch:
blk-cgroup: move blkcg_{get,set}_fc_appid out of line commit db05628435aa
--- drivers/nvme/host/fc.c | 18 ++++++++++++++++++ include/linux/nvme-fc-driver.h | 14 ++++++++++++++ 2 files changed, 32 insertions(+)
drivers/nvme/host/fc.c +18 -0
@@ -1899,6 +1899,24 @@ nvme_fc_ctrl_ioerr_work(struct work_struct *work)nvme_fc_error_recovery (ctrl,"transport detected io error" ); } +/* +* nvme_fc_io_getuuid - Routine called to get the appid field +* associated with request by the lldd +* @req:IO request from nvme fc to driver +* Returns: UUID if there is an appid associated with VM or +* NULL if the user/libvirt has not set the appid to VM +*/ +char *nvme_fc_io_getuuid (struct nvmefc_fcp_req *req) +{ +struct nvme_fc_fcp_op *op =fcp_req_to_fcp_op (req); +struct request *rq = op->rq; + +if (!IS_ENABLED (CONFIG_BLK_CGROUP_FC_APPID ) || !rq->bio) +return NULL ; +return blkcg_get_fc_appid (rq->bio); +} +EXPORT_SYMBOL_GPL (nvme_fc_io_getuuid); +static void nvme_fc_fcpio_done (struct nvmefc_fcp_req *req) {
include/linux/nvme-fc-driver.h +14 -0
@@ -564,6 +564,15 @@ int nvme_fc_rcv_ls_req(struct nvme_fc_remote_port *remoteport, void *lsreqbuf, u32 lsreqbuf_len); +/* +* Routine called to get the appid field associated with request by the lldd +* +* If the return value is NULL : the user/libvirt has not set the appid to VM +* If the return value is non-zero: Returns the appid associated with VM +* +* @req: IO request from nvme fc to driver +*/ +char *nvme_fc_io_getuuid (struct nvmefc_fcp_req *req);/* * *************** LLDD FC-NVME Target/Subsystem API ***************
@@ -1048,5 +1057,10 @@ int nvmet_fc_rcv_fcp_req(struct nvmet_fc_target_port *tgtport,void nvmet_fc_rcv_fcp_abort (struct nvmet_fc_target_port *tgtport,struct nvmefc_tgt_fcp_req *fcpreq); +/* +* add a define, visible to the compiler, that indicates support +* for feature. Allows for conditional compilation in LLDDs. +*/ +#define NVME_FC_FEAT_UUID 0x0001#endif /* _NVME_FC_DRIVER_H */
Signature · 1 line
2.26.2
PATCH v2 2/4 lpfc: commonize VMID code location
In reply to James SmartRemove VMID code from its scsi-specific location and move to a new file solely for VMID code.
--- drivers/scsi/lpfc/Makefile | 2 +- drivers/scsi/lpfc/lpfc_crtn.h | 2 + drivers/scsi/lpfc/lpfc_scsi.c | 256 ------------------------------ drivers/scsi/lpfc/lpfc_vmid.c | 288 ++++++++++++++++++++++++++++++++++ 4 files changed, 291 insertions(+), 257 deletions(-) create mode 100644 drivers/scsi/lpfc/lpfc_vmid.c
drivers/scsi/lpfc/Makefile +1 -1
@@ -33,4 +33,4 @@ obj-$(CONFIG_SCSI_LPFC) := lpfc.o lpfc-objs := lpfc_mem.o lpfc_sli.o lpfc_ct.o lpfc_els.o \ lpfc_hbadisc.o lpfc_init.o lpfc_mbox.o lpfc_nportdisc.o \ lpfc_scsi.o lpfc_attr.o lpfc_vport.o lpfc_debugfs.o lpfc_bsg.o \ - lpfc_nvme.o lpfc_nvmet.o + lpfc_nvme.o lpfc_nvmet.o lpfc_vmid.o
drivers/scsi/lpfc/lpfc_crtn.h +2 -0
@@ -671,6 +671,8 @@ int lpfc_vmid_cmd(struct lpfc_vport *vport,int lpfc_vmid_hash_fn (const char *vmid,int len);struct lpfc_vmid *lpfc_get_vmid_from_hashtable (struct lpfc_vport *vport,uint32_t hash,uint8_t *buf); +int lpfc_vmid_get_appid (struct lpfc_vport *vport,char *uuid,struct +scsi_cmnd * cmd,union lpfc_vmid_io_tag *tag);void lpfc_vmid_vport_cleanup (struct lpfc_vport *vport);int lpfc_issue_els_qfpa (struct lpfc_vport *vport);
drivers/scsi/lpfc/lpfc_scsi.c +0 -256
268 lines not on this page · raw message
drivers/scsi/lpfc/lpfc_vmid.c new file +288 -0
288 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v2 3/4 lpfc: rework lpfc_vmid_get_appid() to be protocol independent
In reply to James SmartRework lpfc_vmid_get_appid() arguments to remove scsi cmd dependency. It's now callable by nvme path.
Fixup scsi calling path for arg change.
--- drivers/scsi/lpfc/lpfc_crtn.h | 5 +++-- drivers/scsi/lpfc/lpfc_scsi.c | 7 ++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/scsi/lpfc/lpfc_crtn.h +3 -2
@@ -671,8 +671,9 @@ int lpfc_vmid_cmd(struct lpfc_vport *vport,int lpfc_vmid_hash_fn (const char *vmid,int len);struct lpfc_vmid *lpfc_get_vmid_from_hashtable (struct lpfc_vport *vport,uint32_t hash,uint8_t *buf); -int lpfc_vmid_get_appid (struct lpfc_vport *vport,char *uuid,struct -scsi_cmnd * cmd,union lpfc_vmid_io_tag *tag); +int lpfc_vmid_get_appid (struct lpfc_vport *vport,char *uuid, +enum dma_data_direction iodir, +union lpfc_vmid_io_tag *tag);void lpfc_vmid_vport_cleanup (struct lpfc_vport *vport);int lpfc_issue_els_qfpa (struct lpfc_vport *vport);
drivers/scsi/lpfc/lpfc_scsi.c +4 -3
@@ -5446,9 +5446,10 @@ lpfc_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *cmnd) uuid =lpfc_is_command_vm_io (cmnd);if (uuid) { - err =lpfc_vmid_get_appid (vport, uuid, cmnd, - (union lpfc_vmid_io_tag *) - &cur_iocbq->vmid_tag); + err =lpfc_vmid_get_appid (vport, uuid, + cmnd->sc_data_direction, + (union lpfc_vmid_io_tag *) + &cur_iocbq->vmid_tag);if (!err) cur_iocbq->cmd_flag |=LPFC_IO_VMID ; }
Signature · 1 line
2.26.2
PATCH v2 4/4 lpfc: Add support for vmid tagging of NVMe I/Os
In reply to James SmartModify the nvme io path to look for vmid support and call the transport to obtain the io's appid value.
--- drivers/scsi/lpfc/lpfc_nvme.c | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
drivers/scsi/lpfc/lpfc_nvme.c +45 -0
@@ -1279,6 +1279,19 @@ lpfc_nvme_prep_io_cmd(struct lpfc_vport *vport,/* Words 13 14 15 are for PBDE support */ +/* add the VMID tags as per switch response */ +if (unlikely (lpfc_ncmd->cur_iocbq.cmd_flag &LPFC_IO_VMID )) { +if (phba->pport->vmid_priority_tagging) { +bf_set (wqe_ccpe, &wqe->fcp_iwrite.wqe_com,1 ); +bf_set (wqe_ccp, &wqe->fcp_iwrite.wqe_com, + lpfc_ncmd->cur_iocbq.vmid_tag.cs_ctl_vmid); + }else { +bf_set (wqe_appid, &wqe->fcp_iwrite.wqe_com,1 ); +bf_set (wqe_wqes, &wqe->fcp_iwrite.wqe_com,1 ); + wqe->words[31 ] = lpfc_ncmd->cur_iocbq.vmid_tag.app_id; + } + } + pwqeq->vport = vport;return 0 ; }
@@ -1504,6 +1517,11 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport,struct lpfc_nvme_fcpreq_priv *freqpriv;struct nvme_common_command *sqe;uint64_t start =0 ; +#if (IS_ENABLED (CONFIG_NVME_FC )) +u8 *uuid =NULL ; +int err; +enum dma_data_direction iodir; +#endif /* Validate pointers. LLDD fault handling with transport does * have timing races.
@@ -1662,6 +1680,33 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport, lpfc_ncmd->ndlp = ndlp; lpfc_ncmd->qidx = lpfc_queue_info->qidx; +#if (IS_ENABLED (CONFIG_NVME_FC )) +/* check the necessary and sufficient condition to support VMID */ +if (lpfc_is_vmid_enabled (phba) && + (ndlp->vmid_support || + phba->pport->vmid_priority_tagging == +LPFC_VMID_PRIO_TAG_ALL_TARGETS )) { +/* is the I/O generated by a VM, get the associated virtual */ +/* entity id */ + uuid =nvme_fc_io_getuuid (pnvme_fcreq); + +if (uuid) { +if (pnvme_fcreq->io_dir ==NVMEFC_FCP_WRITE ) + iodir =DMA_TO_DEVICE ; +else if (pnvme_fcreq->io_dir ==NVMEFC_FCP_READ ) + iodir =DMA_FROM_DEVICE ; +else + iodir =DMA_NONE ; + + err =lpfc_vmid_get_appid (vport, uuid, iodir, + (union lpfc_vmid_io_tag *) + &lpfc_ncmd->cur_iocbq.vmid_tag); +if (!err) + lpfc_ncmd->cur_iocbq.cmd_flag |=LPFC_IO_VMID ; + } + } +#endif +/* * Issue the IO on the WQ indicated by index in the hw_queue_handle. * This identfier was create in our hardware queue create callback
Signature · 1 line
2.26.2
Martin K. Petersen
AppliedJames,
Quoted 1 line
This patch adds vmid support to the nvme-fc transport.
Applied to 5.19/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 10 lines · James Smart
This patch adds vmid support to the nvme-fc transport.
Various virtualization technologies used in Fibre Channel SAN deployments have the ability to identify and associate traffic with specific virtualized applications. The T11 standard defines an application services tag that can be added to FC traffic to aid in identification and monitoring of traffic associated with the applications.
[...]
Applied to 5.19/scsi-queue, thanks!
[1/4] nvme-fc: Add new routine nvme_fc_io_getuuid
https://git.kernel.org/mkp/scsi/c/827fc630e4c8 [2/4] lpfc: commonize VMID code location https://git.kernel.org/mkp/scsi/c/ed913cf4a533 [3/4] lpfc: rework lpfc_vmid_get_appid() to be protocol independent https://git.kernel.org/mkp/scsi/c/348efeca7487 [4/4] lpfc: Add support for vmid tagging of NVMe I/Os https://git.kernel.org/mkp/scsi/c/896325a8b165
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering