Skip to contentSkip to conversation

Introduce support for multiqueue (MQ) in fnic

PATCH Applied Karan Tilak Kumar 2 participants 16 messages Dec 11, 2023 – Dec 19, 2023

Subsystems (2)
lore
Share & tools
lore

Karan Tilak Kumar

author maintainer
#0 lore raw

v6 cover Introduce support for multiqueue (MQ) in fnic

Hi Martin, reviewers,

This cover letter describes the feature: add support for multiqueue (MQ) to fnic driver.

Background: The Virtual Interface Card (VIC) firmware exposes several queues that can be configured for sending IOs and receiving IO responses. Unified Computing System Manager (UCSM) and Intersight Manager (IMM) allows users to configure the number of queues to be used for IOs.

The number of IO queues to be used is stored in a configuration file by the VIC firmware. The fNIC driver reads the configuration file and sets the number of queues to be used. Previously, the driver was hard-coded to use only one queue. With this set of changes, the fNIC driver will configure itself to use multiple queues. This feature takes advantage of the block multiqueue layer to parallelize IOs being sent out of the VIC card.

Here's a brief description of some of the salient patches:

- vnic_scsi.h needs to be in sync with VIC firmware to be able to read the number of queues from the firmware config file. A patch has been created for this.
- In an environment with many fnics (like we see in our customer environments), it is hard to distinguish which fnic is printing logs. Therefore, an fnic number has been included in the logs.
- read the number of queues from the firmware config file.
- include definitions in fnic.h to support multiqueue.
- modify the interrupt service routines (ISRs) to read from the correct registers. The numbers that are used here come from discussions with the VIC firmware team.
- track IO statistics for different queues.
- remove usage of host_lock, and only use fnic_lock in the fnic driver.
- use a hardware queue based spinlock to protect io_req.
- replace the hard-coded zeroth queue with a hardware queue number. This presents a bulk of the changes.
- modify the definition of fnic_queuecommand to accept multiqueue tags.
- improve log messages, and indicate fnic number and multiqueue tags for effective debugging.

Even though the patches have been made into a series, some patches are heavier than others.
But, every effort has been made to keep the purpose of each patch as a single-purpose, and to compile cleanly.

This patchset has been tested as a whole. Therefore, the tested-by fields have been added only to two patches
in the set. All the individual patches compile cleanly. However, I've refrained from adding tested-by to
most of the patches, so as to not mislead the reviewer/reader.

A brief note on the unit tests:

1. Increase number of queues to 64. Load driver. Run IOs via Medusa. 12+ hour run successful.
2. Configure multipathing, and run link flaps on single link. IOs drop briefly, but pick up as expected.
3. Configure multipathing, and run link flaps on two links, with a 30 second delay in between. IOs drop briefly, but pick up as expected.

Repeat the above tests with 1 queue and 32 queues. All tests were successful.

Please consider this patch series for the next merge window.

Changes between v1 and v2:
Suppress a warning raised by a kernel test bot, Incorporate the following review comments from Bart:
Remove outdated comment,
Remove unnecessary out of range tag checks, Remove unnecessary local variable,
Modify function name.

Changes between v2 and v3:
Incorporate review comment from Hannes:
Modify FNIC_MAIN_DBG to prepend fnic number.
Modify FNIC_MAIN_DBG definition to prepend function name and line number.
Modify FNIC_FCS_DBG definition to prepend function name and line number.
Replace FNIC_MAIN_DBG with FNIC_FCS_DBG in fnic_fcs.c Use fnic_num as an argument to FNIC_MAIN_DBG and FNIC_FCS_DBG.
Host number is still used as an argument to FNIC_MAIN_DBG and FNIC_FCS_DBG since it in turn uses shost_printk.
Replace cpy_wq_base with copy_wq_base.
Incorporate the following review comments from Hannes:
Replace cpy_wq_base with copy_wq_base.
Remove C99 style comment.
Extend review comments of FNIC_MAIN_DBG and FNIC_SCSI_DBG to FNIC_ISR_DBG:
Use fnic_num as an argument to FNIC_ISR_DBG. Modify definition of FNIC_ISR_DBG.
Host number is still used as an argument to FNIC_ISR_DBG since it in turn uses shost_printk.
Removed reviewed by tag from Hannes due to additional modifications.
Squash the following commits into one:
scsi: fnic: Remove usage of host_lock
scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand
Incorporate review comment from Hannes:
Replace cpy_wq_base with copy_wq_base.
Incorporate review comment from John Garry:
Replace code in fnic_mq_map_queues_cpus
with blk_mq_pci_map_queues.
Replace shost_printk logs with FNIC_MAIN_DBG. Incorporate the following review comments from Hannes:
Replace cpy_wq_base with copy_wq_base.
Remove hwq as an argument to fnic_queuecommand_int.
Suppress warning from kernel test robot.
Replace new shost_printk comments with FNIC_SCSI_DBG. Replace fnic_queuecommand_int with fnic_queuecommand. Incorporate the following review comment from Hannes:
Use fnic_num as an argument to FNIC_SCSI_DBG. Modify definition of FNIC_SCSI_DBG.
Host number is still an argument since
FNIC_SCSI_DBG in turn uses shost_printk.
Create a separate patch to increment driver version. Increment driver version number to 1.7.0.0.

Changes between v3 and v4:
Incorporate review comments from Martin and Hannes:
Undo the change to replace host number with fnic number in debugfs since kernel stack uses host number. Use ida_alloc to allocate ID for fnic number.

Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v5 and v6:
Incorporate comments from Martin:
Rebase changes to 6.8/scsi-queue,
Resolve merge conflicts.

Thanks and regards,
Karan

Karan Tilak Kumar (13):
scsi: fnic: Modify definitions to sync with VIC firmware
scsi: fnic: Add and use fnic number
scsi: fnic: Add and improve log messages
scsi: fnic: Rename wq_copy to hw_copy_wq
scsi: fnic: Get copy workqueue count and interrupt mode from config
scsi: fnic: Refactor and redefine fnic.h for multiqueue
scsi: fnic: Modify ISRs to support multiqueue(MQ)
scsi: fnic: Define stats to track multiqueue (MQ) IOs
scsi: fnic: Remove usage of host_lock
scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c
scsi: fnic: Add support for multiqueue (MQ) in fnic driver
scsi: fnic: Improve logs and add support for multiqueue (MQ)
scsi: fnic: Increment driver version

 drivers/scsi/fnic/fnic.h       |  68 ++-
 drivers/scsi/fnic/fnic_fcs.c   |  63 +--
 drivers/scsi/fnic/fnic_isr.c   | 168 +++++--
 drivers/scsi/fnic/fnic_main.c  | 144 ++++--
 drivers/scsi/fnic/fnic_res.c   |  48 +-
 drivers/scsi/fnic/fnic_scsi.c  | 868 ++++++++++++++++++---------------
 drivers/scsi/fnic/fnic_stats.h |   3 +
 drivers/scsi/fnic/fnic_trace.c |  11 +
 drivers/scsi/fnic/vnic_dev.c   |   4 +
 drivers/scsi/fnic/vnic_scsi.h  |  13 +-
 10 files changed, 847 insertions(+), 543 deletions(-)
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH v6 1/13 scsi: fnic: Modify definitions to sync with VIC firmware

VIC firmware has updated definitions.
Modify structure and definitions to sync with the latest VIC firmware.

---
 drivers/scsi/fnic/vnic_scsi.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/scsi/fnic/vnic_scsi.h +11 -2
index 4e12f7b32d9d..f715f7942bfe 100644
@@ -26,7 +26,7 @@
 #define VNIC_FNIC_RATOV_MAX                 255000
 
 #define VNIC_FNIC_MAXDATAFIELDSIZE_MIN      256
-#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX      2112
+#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX      2048
 
 #define VNIC_FNIC_FLOGI_RETRIES_MIN         0
 #define VNIC_FNIC_FLOGI_RETRIES_MAX         0xffffffff
@@ -55,7 +55,7 @@
 #define VNIC_FNIC_PORT_DOWN_IO_RETRIES_MAX  255
 
 #define VNIC_FNIC_LUNS_PER_TARGET_MIN       1
-#define VNIC_FNIC_LUNS_PER_TARGET_MAX       1024
+#define VNIC_FNIC_LUNS_PER_TARGET_MAX       4096
 
 /* Device-specific region: scsi configuration */
 struct vnic_fc_config {
@@ -79,10 +79,19 @@ struct vnic_fc_config {
 	u16 ra_tov;
 	u16 intr_timer;
 	u8 intr_timer_type;
+	u8 intr_mode;
+	u8 lun_queue_depth;
+	u8 io_timeout_retry;
+	u16 wq_copy_count;
 };
 
 #define VFCF_FCP_SEQ_LVL_ERR	0x1	/* Enable FCP-2 Error Recovery */
 #define VFCF_PERBI		0x2	/* persistent binding info available */
 #define VFCF_FIP_CAPABLE	0x4	/* firmware can handle FIP */
 
+#define VFCF_FC_INITIATOR         0x20    /* FC Initiator Mode */
+#define VFCF_FC_TARGET            0x40    /* FC Target Mode */
+#define VFCF_FC_NVME_INITIATOR    0x80    /* FC-NVMe Initiator Mode */
+#define VFCF_FC_NVME_TARGET       0x100   /* FC-NVMe Target Mode */
+
 #endif /* _VNIC_SCSI_H_ */
Signature · 1 line
2.31.1
author maintainer
#2 lore raw

PATCH v6 2/13 scsi: fnic: Add and use fnic number

In reply to Karan Tilak Kumar

Add fnic_num in fnic.h to identify fnic in a multi-fnic environment. Increment and set the fnic number during driver load in fnic_probe. Replace the host number with fnic number in debugfs.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v3 and v4:
Incorporate review comments from Martin and Hannes:
Undo the change to replace host number with fnic number in debugfs since kernel stack uses host number. Use ida_alloc to allocate ID for fnic number.

---
 drivers/scsi/fnic/fnic.h      |  1 +
 drivers/scsi/fnic/fnic_main.c | 18 ++++++++++++++++--
 2 files changed, 17 insertions(+), 2 deletions(-)
drivers/scsi/fnic/fnic.h +1 -0
index 22cef283b2b9..721a31a03628 100644
@@ -216,6 +216,7 @@ struct fnic_event {
 
 /* Per-instance private data structure */
 struct fnic {
+	int fnic_num;
 	struct fc_lport *lport;
 	struct fcoe_ctlr ctlr;		/* FIP FCoE controller structure */
 	struct vnic_dev_bar bar0;
drivers/scsi/fnic/fnic_main.c +16 -2
index f27f9319e0b2..3044723e125e 100644
@@ -39,6 +39,7 @@ static struct kmem_cache *fnic_sgl_cache[FNIC_SGL_NUM_CACHES];
 static struct kmem_cache *fnic_io_req_cache;
 static LIST_HEAD(fnic_list);
 static DEFINE_SPINLOCK(fnic_list_lock);
+static DEFINE_IDA(fnic_ida);
 
 /* Supported devices by fnic module */
 static struct pci_device_id fnic_id_table[] = {
@@ -583,7 +584,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct fc_lport *lp;
 	struct fnic *fnic;
 	mempool_t *pool;
-	int err;
+	int err = 0;
+	int fnic_id = 0;
 	int i;
 	unsigned long flags;
 
@@ -597,8 +599,16 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		err = -ENOMEM;
 		goto err_out;
 	}
+
 	host = lp->host;
 	fnic = lport_priv(lp);
+
+	fnic_id = ida_alloc(&fnic_ida, GFP_KERNEL);
+	if (fnic_id < 0) {
+		pr_err("Unable to alloc fnic ID\n");
+		err = fnic_id;
+		goto err_out_ida_alloc;
+	}
 	fnic->lport = lp;
 	fnic->ctlr.lp = lp;
 
@@ -608,7 +618,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		 host->host_no);
 
 	host->transportt = fnic_fc_transport;
-
+	fnic->fnic_num = fnic_id;
 	fnic_stats_debugfs_init(fnic);
 
 	/* Setup PCI resources */
@@ -951,6 +961,8 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	pci_disable_device(pdev);
 err_out_free_hba:
 	fnic_stats_debugfs_remove(fnic);
+	ida_free(&fnic_ida, fnic->fnic_num);
+err_out_ida_alloc:
 	scsi_host_put(lp->host);
 err_out:
 	return err;
@@ -1031,6 +1043,7 @@ static void fnic_remove(struct pci_dev *pdev)
 	fnic_iounmap(fnic);
 	pci_release_regions(pdev);
 	pci_disable_device(pdev);
+	ida_free(&fnic_ida, fnic->fnic_num);
 	scsi_host_put(lp->host);
 }
 
@@ -1168,6 +1181,7 @@ static void __exit fnic_cleanup_module(void)
 	fnic_trace_free();
 	fnic_fc_trace_free();
 	fnic_debugfs_terminate();
+	ida_destroy(&fnic_ida);
 }
 
 module_init(fnic_init_module);
Signature · 1 line
2.31.1
author maintainer
#3 lore raw

PATCH v6 3/13 scsi: fnic: Add and improve log messages

In reply to Karan Tilak Kumar

Add link related log messages in fnic_fcs.c, Improve log message in fnic_fcs.c,
Add log message in vnic_dev.c.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate review comment from Hannes:
Modify FNIC_MAIN_DBG to prepend fnic number.
Modify FNIC_MAIN_DBG definition to prepend function name and line number.
Modify FNIC_FCS_DBG definition to prepend function name and line number.
Replace FNIC_MAIN_DBG with FNIC_FCS_DBG in fnic_fcs.c Use fnic_num as an argument to FNIC_MAIN_DBG and FNIC_FCS_DBG. Host number is still used as an argument to FNIC_MAIN_DBG and FNIC_FCS_DBG since it in turn uses shost_printk.

---
 drivers/scsi/fnic/fnic.h      | 12 ++++---
 drivers/scsi/fnic/fnic_fcs.c  | 63 +++++++++++++++++++----------------
 drivers/scsi/fnic/fnic_main.c |  4 +--
 drivers/scsi/fnic/vnic_dev.c  |  4 +++
 4 files changed, 49 insertions(+), 34 deletions(-)
drivers/scsi/fnic/fnic.h +8 -4
index 721a31a03628..4dde8afbb69b 100644
@@ -144,13 +144,17 @@ do {								\
 		} while (0);					\
 } while (0)
 
-#define FNIC_MAIN_DBG(kern_level, host, fmt, args...)		\
+#define FNIC_MAIN_DBG(kern_level, host, fnic_num, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_MAIN_LOGGING,			\
-			 shost_printk(kern_level, host, fmt, ##args);)
+			 shost_printk(kern_level, host,			\
+				"fnic<%d>: %s: %d: " fmt, fnic_num,\
+				__func__, __LINE__, ##args);)
 
-#define FNIC_FCS_DBG(kern_level, host, fmt, args...)		\
+#define FNIC_FCS_DBG(kern_level, host, fnic_num, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_FCS_LOGGING,			\
-			 shost_printk(kern_level, host, fmt, ##args);)
+			 shost_printk(kern_level, host,			\
+				"fnic<%d>: %s: %d: " fmt, fnic_num,\
+				__func__, __LINE__, ##args);)
 
 #define FNIC_SCSI_DBG(kern_level, host, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_SCSI_LOGGING,			\
drivers/scsi/fnic/fnic_fcs.c +35 -28

200 lines not on this page · raw message

drivers/scsi/fnic/fnic_main.c +2 -2
index 3044723e125e..cc20c12012e3 100644
@@ -210,7 +210,7 @@ static struct fc_host_statistics *fnic_get_stats(struct Scsi_Host *host)
 	spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 
 	if (ret) {
-		FNIC_MAIN_DBG(KERN_DEBUG, fnic->lport->host,
+		FNIC_MAIN_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num,
 			      "fnic: Get vnic stats failed"
 			      " 0x%x", ret);
 		return stats;
@@ -322,7 +322,7 @@ static void fnic_reset_host_stats(struct Scsi_Host *host)
 	spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 
 	if (ret) {
-		FNIC_MAIN_DBG(KERN_DEBUG, fnic->lport->host,
+		FNIC_MAIN_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num,
 				"fnic: Reset vnic stats failed"
 				" 0x%x", ret);
 		return;
drivers/scsi/fnic/vnic_dev.c +4 -0
index 3e5b437c0492..e0b173cc9d5f 100644
@@ -143,6 +143,10 @@ static int vnic_dev_discover_res(struct vnic_dev *vdev,
 		vdev->res[type].vaddr = (char __iomem *)bar->vaddr + bar_offset;
 	}
 
+	pr_info("res_type_wq: %d res_type_rq: %d res_type_cq: %d res_type_intr_ctrl: %d\n",
+		vdev->res[RES_TYPE_WQ].count, vdev->res[RES_TYPE_RQ].count,
+		vdev->res[RES_TYPE_CQ].count, vdev->res[RES_TYPE_INTR_CTRL].count);
+
 	return 0;
 }
 
Signature · 1 line
2.31.1
author maintainer
#4 lore raw

PATCH v6 4/13 scsi: fnic: Rename wq_copy to hw_copy_wq

In reply to Karan Tilak Kumar

Rename wq_copy to hw_copy_wq to accurately describe the copy workqueue. This will also help distinguish this data structure from software data structures that can be introduced.

---
 drivers/scsi/fnic/fnic.h      |  2 +-
 drivers/scsi/fnic/fnic_isr.c  |  2 +-
 drivers/scsi/fnic/fnic_main.c |  8 ++++----
 drivers/scsi/fnic/fnic_res.c  |  6 +++---
 drivers/scsi/fnic/fnic_scsi.c | 12 ++++++------
 5 files changed, 15 insertions(+), 15 deletions(-)
drivers/scsi/fnic/fnic.h +1 -1
index 4dde8afbb69b..2e68209181a8 100644
@@ -307,7 +307,7 @@ struct fnic {
 	/*** FIP related data members  -- end ***/
 
 	/* copy work queue cache line section */
-	____cacheline_aligned struct vnic_wq_copy wq_copy[FNIC_WQ_COPY_MAX];
+	____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX];
 	/* completion queue cache line section */
 	____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX];
 
drivers/scsi/fnic/fnic_isr.c +1 -1
index 8896758fed8c..dff9689023e4 100644
@@ -224,7 +224,7 @@ int fnic_set_intr_mode(struct fnic *fnic)
 {
 	unsigned int n = ARRAY_SIZE(fnic->rq);
 	unsigned int m = ARRAY_SIZE(fnic->wq);
-	unsigned int o = ARRAY_SIZE(fnic->wq_copy);
+	unsigned int o = ARRAY_SIZE(fnic->hw_copy_wq);
 
 	/*
 	 * Set interrupt mode (INTx, MSI, MSI-X) depending
drivers/scsi/fnic/fnic_main.c +4 -4
index cc20c12012e3..dacefbeb44f0 100644
@@ -355,7 +355,7 @@ void fnic_log_q_error(struct fnic *fnic)
 	}
 
 	for (i = 0; i < fnic->wq_copy_count; i++) {
-		error_status = ioread32(&fnic->wq_copy[i].ctrl->error_status);
+		error_status = ioread32(&fnic->hw_copy_wq[i].ctrl->error_status);
 		if (error_status)
 			shost_printk(KERN_ERR, fnic->lport->host,
 				     "CWQ[%d] error_status"
@@ -492,7 +492,7 @@ static int fnic_cleanup(struct fnic *fnic)
 			return err;
 	}
 	for (i = 0; i < fnic->wq_copy_count; i++) {
-		err = vnic_wq_copy_disable(&fnic->wq_copy[i]);
+		err = vnic_wq_copy_disable(&fnic->hw_copy_wq[i]);
 		if (err)
 			return err;
 	}
@@ -508,7 +508,7 @@ static int fnic_cleanup(struct fnic *fnic)
 	for (i = 0; i < fnic->rq_count; i++)
 		vnic_rq_clean(&fnic->rq[i], fnic_free_rq_buf);
 	for (i = 0; i < fnic->wq_copy_count; i++)
-		vnic_wq_copy_clean(&fnic->wq_copy[i],
+		vnic_wq_copy_clean(&fnic->hw_copy_wq[i],
 				   fnic_wq_copy_cleanup_handler);
 
 	for (i = 0; i < fnic->cq_count; i++)
@@ -909,7 +909,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	for (i = 0; i < fnic->raw_wq_count; i++)
 		vnic_wq_enable(&fnic->wq[i]);
 	for (i = 0; i < fnic->wq_copy_count; i++)
-		vnic_wq_copy_enable(&fnic->wq_copy[i]);
+		vnic_wq_copy_enable(&fnic->hw_copy_wq[i]);
 
 	fc_fabric_login(lp);
 
drivers/scsi/fnic/fnic_res.c +3 -3
index a1c9cfcace7f..109316cc4ad9 100644
@@ -203,7 +203,7 @@ void fnic_free_vnic_resources(struct fnic *fnic)
 		vnic_wq_free(&fnic->wq[i]);
 
 	for (i = 0; i < fnic->wq_copy_count; i++)
-		vnic_wq_copy_free(&fnic->wq_copy[i]);
+		vnic_wq_copy_free(&fnic->hw_copy_wq[i]);
 
 	for (i = 0; i < fnic->rq_count; i++)
 		vnic_rq_free(&fnic->rq[i]);
@@ -250,7 +250,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic)
 
 	/* Allocate Copy WQs used for SCSI IOs */
 	for (i = 0; i < fnic->wq_copy_count; i++) {
-		err = vnic_wq_copy_alloc(fnic->vdev, &fnic->wq_copy[i],
+		err = vnic_wq_copy_alloc(fnic->vdev, &fnic->hw_copy_wq[i],
 			(fnic->raw_wq_count + i),
 			fnic->config.wq_copy_desc_count,
 			sizeof(struct fcpio_host_req));
@@ -357,7 +357,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic)
 	}
 
 	for (i = 0; i < fnic->wq_copy_count; i++) {
-		vnic_wq_copy_init(&fnic->wq_copy[i],
+		vnic_wq_copy_init(&fnic->hw_copy_wq[i],
 				  0 /* cq_index 0 - always */,
 				  error_interrupt_enable,
 				  error_interrupt_offset);
drivers/scsi/fnic/fnic_scsi.c +6 -6
index 416d81954819..b7dc304446f8 100644
@@ -193,7 +193,7 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags,
  */
 int fnic_fw_reset_handler(struct fnic *fnic)
 {
-	struct vnic_wq_copy *wq = &fnic->wq_copy[0];
+	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
 	int ret = 0;
 	unsigned long flags;
 
@@ -246,7 +246,7 @@ int fnic_fw_reset_handler(struct fnic *fnic)
  */
 int fnic_flogi_reg_handler(struct fnic *fnic, u32 fc_id)
 {
-	struct vnic_wq_copy *wq = &fnic->wq_copy[0];
+	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
 	enum fcpio_flogi_reg_format_type format;
 	struct fc_lport *lp = fnic->lport;
 	u8 gw_mac[ETH_ALEN];
@@ -551,7 +551,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 	fnic_priv(sc)->flags |= FNIC_IO_INITIALIZED;
 
 	/* create copy wq desc and enqueue it */
-	wq = &fnic->wq_copy[0];
+	wq = &fnic->hw_copy_wq[0];
 	ret = fnic_queue_wq_copy_desc(fnic, wq, io_req, sc, sg_count);
 	if (ret) {
 		/*
@@ -782,7 +782,7 @@ static inline void fnic_fcpio_ack_handler(struct fnic *fnic,
 	u64 *ox_id_tag = (u64 *)(void *)desc;
 
 	/* mark the ack state */
-	wq = &fnic->wq_copy[cq_index - fnic->raw_wq_count - fnic->rq_count];
+	wq = &fnic->hw_copy_wq[cq_index - fnic->raw_wq_count - fnic->rq_count];
 	spin_lock_irqsave(&fnic->wq_copy_lock[0], flags);
 
 	fnic->fnic_stats.misc_stats.last_ack_time = jiffies;
@@ -1502,7 +1502,7 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag,
 					  u32 task_req, u8 *fc_lun,
 					  struct fnic_io_req *io_req)
 {
-	struct vnic_wq_copy *wq = &fnic->wq_copy[0];
+	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
 	struct Scsi_Host *host = fnic->lport->host;
 	struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;
 	unsigned long flags;
@@ -1948,7 +1948,7 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic,
 				       struct scsi_cmnd *sc,
 				       struct fnic_io_req *io_req)
 {
-	struct vnic_wq_copy *wq = &fnic->wq_copy[0];
+	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
 	struct Scsi_Host *host = fnic->lport->host;
 	struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;
 	struct scsi_lun fc_lun;
Signature · 1 line
2.31.1
author maintainer
#5 lore raw

PATCH v6 5/13 scsi: fnic: Get copy workqueue count and interrupt mode from config

In reply to Karan Tilak Kumar

Get the copy workqueue count and interrupt mode from the configuration. The config can be changed via UCSM. Add logs to print the interrupt mode and copy workqueue count. Add logs to print the vNIC resources.

---
 drivers/scsi/fnic/fnic_res.c | 42 ++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)
drivers/scsi/fnic/fnic_res.c +36 -6
index 109316cc4ad9..33dd27f6f24e 100644
@@ -57,6 +57,8 @@ int fnic_get_vnic_config(struct fnic *fnic)
 	GET_CONFIG(port_down_timeout);
 	GET_CONFIG(port_down_io_retries);
 	GET_CONFIG(luns_per_tgt);
+	GET_CONFIG(intr_mode);
+	GET_CONFIG(wq_copy_count);
 
 	c->wq_enet_desc_count =
 		min_t(u32, VNIC_FNIC_WQ_DESCS_MAX,
@@ -131,6 +133,12 @@ int fnic_get_vnic_config(struct fnic *fnic)
 	c->intr_timer = min_t(u16, VNIC_INTR_TIMER_MAX, c->intr_timer);
 	c->intr_timer_type = c->intr_timer_type;
 
+	/* for older firmware, GET_CONFIG will not return anything */
+	if (c->wq_copy_count == 0)
+		c->wq_copy_count = 1;
+
+	c->wq_copy_count = min_t(u16, FNIC_WQ_COPY_MAX, c->wq_copy_count);
+
 	shost_printk(KERN_INFO, fnic->lport->host,
 		     "vNIC MAC addr %pM "
 		     "wq/wq_copy/rq %d/%d/%d\n",
@@ -161,6 +169,10 @@ int fnic_get_vnic_config(struct fnic *fnic)
 	shost_printk(KERN_INFO, fnic->lport->host,
 		     "vNIC port dn io retries %d port dn timeout %d\n",
 		     c->port_down_io_retries, c->port_down_timeout);
+	shost_printk(KERN_INFO, fnic->lport->host,
+			"vNIC wq_copy_count: %d\n", c->wq_copy_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+			"vNIC intr mode: %d\n", c->intr_mode);
 
 	return 0;
 }
@@ -187,12 +199,25 @@ int fnic_set_nic_config(struct fnic *fnic, u8 rss_default_cpu,
 void fnic_get_res_counts(struct fnic *fnic)
 {
 	fnic->wq_count = vnic_dev_get_res_count(fnic->vdev, RES_TYPE_WQ);
-	fnic->raw_wq_count = fnic->wq_count - 1;
-	fnic->wq_copy_count = fnic->wq_count - fnic->raw_wq_count;
+	fnic->raw_wq_count = 1;
+	fnic->wq_copy_count = fnic->config.wq_copy_count;
 	fnic->rq_count = vnic_dev_get_res_count(fnic->vdev, RES_TYPE_RQ);
 	fnic->cq_count = vnic_dev_get_res_count(fnic->vdev, RES_TYPE_CQ);
 	fnic->intr_count = vnic_dev_get_res_count(fnic->vdev,
 		RES_TYPE_INTR_CTRL);
+
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources wq_count: %d\n", fnic->wq_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources raw_wq_count: %d\n", fnic->raw_wq_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources wq_copy_count: %d\n", fnic->wq_copy_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources rq_count: %d\n", fnic->rq_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources cq_count: %d\n", fnic->cq_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+		"vNIC fw resources intr_count: %d\n", fnic->intr_count);
 }
 
 void fnic_free_vnic_resources(struct fnic *fnic)
@@ -234,10 +259,15 @@ int fnic_alloc_vnic_resources(struct fnic *fnic)
 		     intr_mode == VNIC_DEV_INTR_MODE_MSIX ?
 		     "MSI-X" : "unknown");
 
-	shost_printk(KERN_INFO, fnic->lport->host, "vNIC resources avail: "
-		     "wq %d cp_wq %d raw_wq %d rq %d cq %d intr %d\n",
-		     fnic->wq_count, fnic->wq_copy_count, fnic->raw_wq_count,
-		     fnic->rq_count, fnic->cq_count, fnic->intr_count);
+	shost_printk(KERN_INFO, fnic->lport->host,
+			"vNIC resources avail: wq %d cp_wq %d raw_wq %d rq %d",
+			fnic->wq_count, fnic->wq_copy_count,
+			fnic->raw_wq_count, fnic->rq_count);
+
+	shost_printk(KERN_INFO, fnic->lport->host,
+			"vNIC resources avail: cq %d intr %d cpy-wq desc count %d\n",
+			fnic->cq_count, fnic->intr_count,
+			fnic->config.wq_copy_desc_count);
 
 	/* Allocate Raw WQ used for FCS frames */
 	for (i = 0; i < fnic->raw_wq_count; i++) {
Signature · 1 line
2.31.1
author maintainer
#6 lore raw

PATCH v6 6/13 scsi: fnic: Refactor and redefine fnic.h for multiqueue

In reply to Karan Tilak Kumar

Refactor and re-define values in fnic.h to implement multiqueue(MQ) functionality.

VIC firmware allows fnic to create up to 64 copy workqueues. Update the copy workqueue max to 64. Modify the interrupt index to be in sync with the firmware to support MQ.
Add irq number to the MSIX entry.
Define a software workqueue table to track the status of io_reqs. Define a base for the copy workqueue.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Replace cpy_wq_base with copy_wq_base.

---
 drivers/scsi/fnic/fnic.h | 30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)
drivers/scsi/fnic/fnic.h +22 -8
index 2e68209181a8..47173f1eec1e 100644
@@ -167,12 +167,21 @@ do {								\
 #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...)          \
 	shost_printk(kern_level, host, fmt, ##args)
 
+#define FNIC_WQ_COPY_MAX 64
+#define FNIC_WQ_MAX 1
+#define FNIC_RQ_MAX 1
+#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX)
+#define FNIC_DFLT_IO_COMPLETIONS 256
+
+#define FNIC_MQ_CQ_INDEX        2
+
 extern const char *fnic_state_str[];
 
 enum fnic_intx_intr_index {
 	FNIC_INTX_WQ_RQ_COPYWQ,
-	FNIC_INTX_ERR,
+	FNIC_INTX_DUMMY,
 	FNIC_INTX_NOTIFY,
+	FNIC_INTX_ERR,
 	FNIC_INTX_INTR_MAX,
 };
 
@@ -180,7 +189,7 @@ enum fnic_msix_intr_index {
 	FNIC_MSIX_RQ,
 	FNIC_MSIX_WQ,
 	FNIC_MSIX_WQ_COPY,
-	FNIC_MSIX_ERR_NOTIFY,
+	FNIC_MSIX_ERR_NOTIFY = FNIC_MSIX_WQ_COPY + FNIC_WQ_COPY_MAX,
 	FNIC_MSIX_INTR_MAX,
 };
 
@@ -189,6 +198,7 @@ struct fnic_msix_entry {
 	char devname[IFNAMSIZ + 11];
 	irqreturn_t (*isr)(int, void *);
 	void *devid;
+	int irq_num;
 };
 
 enum fnic_state {
@@ -198,12 +208,6 @@ enum fnic_state {
 	FNIC_IN_ETH_TRANS_FC_MODE,
 };
 
-#define FNIC_WQ_COPY_MAX 1
-#define FNIC_WQ_MAX 1
-#define FNIC_RQ_MAX 1
-#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX)
-#define FNIC_DFLT_IO_COMPLETIONS 256
-
 struct mempool;
 
 enum fnic_evt {
@@ -218,6 +222,13 @@ struct fnic_event {
 	enum fnic_evt event;
 };
 
+struct fnic_cpy_wq {
+	unsigned long hw_lock_flags;
+	u16 active_ioreq_count;
+	u16 ioreq_table_size;
+	____cacheline_aligned struct fnic_io_req **io_req_table;
+};
+
 /* Per-instance private data structure */
 struct fnic {
 	int fnic_num;
@@ -289,6 +300,7 @@ struct fnic {
 	mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES];
 	spinlock_t io_req_lock[FNIC_IO_LOCKS];	/* locks for scsi cmnds */
 
+	unsigned int copy_wq_base;
 	struct work_struct link_work;
 	struct work_struct frame_work;
 	struct sk_buff_head frame_queue;
@@ -308,6 +320,8 @@ struct fnic {
 
 	/* copy work queue cache line section */
 	____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX];
+	____cacheline_aligned struct fnic_cpy_wq sw_copy_wq[FNIC_WQ_COPY_MAX];
+
 	/* completion queue cache line section */
 	____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX];
 
Signature · 1 line
2.31.1
author maintainer
#7 lore raw

PATCH v6 7/13 scsi: fnic: Modify ISRs to support multiqueue(MQ)

In reply to Karan Tilak Kumar

Modify interrupt service routines for INTx, MSI, and MSI-x to support multiqueue.
Modify parameter list of fnic_wq_copy_cmpl_handler to take cq_index.
Modify fnic_cleanup function to use the new function call of fnic_wq_copy_cmpl_handler. Refactor code to set
interrupt mode to MSI-x to a new function.
Add a new stat for intx_dummy.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate the following review comments from Hannes:
Replace cpy_wq_base with copy_wq_base. Remove C99 style comment.
Extend review comments of FNIC_MAIN_DBG and FNIC_SCSI_DBG to FNIC_ISR_DBG:
Use fnic_num as an argument to FNIC_ISR_DBG. Modify definition of FNIC_ISR_DBG.
Host number is still used as an argument to FNIC_ISR_DBG since it in turn uses shost_printk.
Removed reviewed by tag from Hannes due to additional modifications.

Changes between v1 and v2:
Suppress warning from kernel test bot.

---
 drivers/scsi/fnic/fnic.h       |   9 +-
 drivers/scsi/fnic/fnic_isr.c   | 166 ++++++++++++++++++++++++---------
 drivers/scsi/fnic/fnic_main.c  |   4 +-
 drivers/scsi/fnic/fnic_scsi.c  |  38 +++-----
 drivers/scsi/fnic/fnic_stats.h |   1 +
 5 files changed, 144 insertions(+), 74 deletions(-)
drivers/scsi/fnic/fnic.h +6 -3
index 47173f1eec1e..8b3e4fe1ce92 100644
@@ -160,9 +160,11 @@ do {								\
 	FNIC_CHECK_LOGGING(FNIC_SCSI_LOGGING,			\
 			 shost_printk(kern_level, host, fmt, ##args);)
 
-#define FNIC_ISR_DBG(kern_level, host, fmt, args...)		\
+#define FNIC_ISR_DBG(kern_level, host, fnic_num, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_ISR_LOGGING,			\
-			 shost_printk(kern_level, host, fmt, ##args);)
+			 shost_printk(kern_level, host,			\
+				"fnic<%d>: %s: %d: " fmt, fnic_num,\
+				__func__, __LINE__, ##args);)
 
 #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...)          \
 	shost_printk(kern_level, host, fmt, ##args)
@@ -349,6 +351,7 @@ extern const struct attribute_group *fnic_host_groups[];
 
 void fnic_clear_intr_mode(struct fnic *fnic);
 int fnic_set_intr_mode(struct fnic *fnic);
+int fnic_set_intr_mode_msix(struct fnic *fnic);
 void fnic_free_intr(struct fnic *fnic);
 int fnic_request_intr(struct fnic *fnic);
 
@@ -375,7 +378,7 @@ void fnic_scsi_cleanup(struct fc_lport *);
 void fnic_scsi_abort_io(struct fc_lport *);
 void fnic_empty_scsi_cleanup(struct fc_lport *);
 void fnic_exch_mgr_reset(struct fc_lport *, u32, u32);
-int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int);
+int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do, unsigned int cq_index);
 int fnic_wq_cmpl_handler(struct fnic *fnic, int);
 int fnic_flogi_reg_handler(struct fnic *fnic, u32);
 void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq,
drivers/scsi/fnic/fnic_isr.c +120 -46

238 lines not on this page · raw message

drivers/scsi/fnic/fnic_main.c +3 -1
index dacefbeb44f0..5726c0fb6f0f 100644
@@ -476,6 +476,7 @@ static int fnic_cleanup(struct fnic *fnic)
 {
 	unsigned int i;
 	int err;
+	int raw_wq_rq_counts;
 
 	vnic_dev_disable(fnic->vdev);
 	for (i = 0; i < fnic->intr_count; i++)
@@ -495,10 +496,11 @@ static int fnic_cleanup(struct fnic *fnic)
 		err = vnic_wq_copy_disable(&fnic->hw_copy_wq[i]);
 		if (err)
 			return err;
+		raw_wq_rq_counts = fnic->raw_wq_count + fnic->rq_count;
+		fnic_wq_copy_cmpl_handler(fnic, -1, i + raw_wq_rq_counts);
 	}
 
 	/* Clean up completed IOs and FCS frames */
-	fnic_wq_copy_cmpl_handler(fnic, io_completions);
 	fnic_wq_cmpl_handler(fnic, -1);
 	fnic_rq_cmpl_handler(fnic, -1);
 
drivers/scsi/fnic/fnic_scsi.c +14 -24
index b7dc304446f8..c7f3bfc650d1 100644
@@ -1319,10 +1319,8 @@ static int fnic_fcpio_cmpl_handler(struct vnic_dev *vdev,
  * fnic_wq_copy_cmpl_handler
  * Routine to process wq copy
  */
-int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do)
+int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do, unsigned int cq_index)
 {
-	unsigned int wq_work_done = 0;
-	unsigned int i, cq_index;
 	unsigned int cur_work_done;
 	struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;
 	u64 start_jiffies = 0;
@@ -1330,28 +1328,20 @@ int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do)
 	u64 delta_jiffies = 0;
 	u64 delta_ms = 0;
 
-	for (i = 0; i < fnic->wq_copy_count; i++) {
-		cq_index = i + fnic->raw_wq_count + fnic->rq_count;
-
-		start_jiffies = jiffies;
-		cur_work_done = vnic_cq_copy_service(&fnic->cq[cq_index],
-						     fnic_fcpio_cmpl_handler,
-						     copy_work_to_do);
-		end_jiffies = jiffies;
-
-		wq_work_done += cur_work_done;
-		delta_jiffies = end_jiffies - start_jiffies;
-		if (delta_jiffies >
-			(u64) atomic64_read(&misc_stats->max_isr_jiffies)) {
-			atomic64_set(&misc_stats->max_isr_jiffies,
-					delta_jiffies);
-			delta_ms = jiffies_to_msecs(delta_jiffies);
-			atomic64_set(&misc_stats->max_isr_time_ms, delta_ms);
-			atomic64_set(&misc_stats->corr_work_done,
-					cur_work_done);
-		}
+	start_jiffies = jiffies;
+	cur_work_done = vnic_cq_copy_service(&fnic->cq[cq_index],
+					fnic_fcpio_cmpl_handler,
+					copy_work_to_do);
+	end_jiffies = jiffies;
+	delta_jiffies = end_jiffies - start_jiffies;
+	if (delta_jiffies > (u64) atomic64_read(&misc_stats->max_isr_jiffies)) {
+		atomic64_set(&misc_stats->max_isr_jiffies, delta_jiffies);
+		delta_ms = jiffies_to_msecs(delta_jiffies);
+		atomic64_set(&misc_stats->max_isr_time_ms, delta_ms);
+		atomic64_set(&misc_stats->corr_work_done, cur_work_done);
 	}
-	return wq_work_done;
+
+	return cur_work_done;
 }
 
 static bool fnic_cleanup_io_iter(struct scsi_cmnd *sc, void *data)
drivers/scsi/fnic/fnic_stats.h +1 -0
index bdf639eef8cf..07d1556e3c32 100644
@@ -103,6 +103,7 @@ struct misc_stats {
 	atomic64_t rport_not_ready;
 	atomic64_t frame_errors;
 	atomic64_t current_port_speed;
+	atomic64_t intx_dummy;
 };
 
 struct fnic_stats {
Signature · 1 line
2.31.1
author maintainer
#8 lore raw

PATCH v6 8/13 scsi: fnic: Define stats to track multiqueue (MQ) IOs

In reply to Karan Tilak Kumar

Define an array to track IOs for the different queues, print the IO stats in fnic get stats data.

---
 drivers/scsi/fnic/fnic_stats.h |  2 ++
 drivers/scsi/fnic/fnic_trace.c | 11 +++++++++++
 2 files changed, 13 insertions(+)
drivers/scsi/fnic/fnic_stats.h +2 -0
index 07d1556e3c32..9d7f98c452dd 100644
@@ -2,6 +2,7 @@
 /* Copyright 2013 Cisco Systems, Inc.  All rights reserved. */
 #ifndef _FNIC_STATS_H_
 #define _FNIC_STATS_H_
+#define FNIC_MQ_MAX_QUEUES 64
 
 struct stats_timestamps {
 	struct timespec64 last_reset_time;
@@ -26,6 +27,7 @@ struct io_path_stats {
 	atomic64_t io_btw_10000_to_30000_msec;
 	atomic64_t io_greater_than_30000_msec;
 	atomic64_t current_max_io_time;
+	atomic64_t ios[FNIC_MQ_MAX_QUEUES];
 };
 
 struct abort_stats {
drivers/scsi/fnic/fnic_trace.c +11 -0
index be0d7c57b242..aaa4ea02fb7c 100644
@@ -204,6 +204,7 @@ int fnic_get_stats_data(struct stats_debug_info *debug,
 	int len = 0;
 	int buf_size = debug->buf_size;
 	struct timespec64 val1, val2;
+	int i = 0;
 
 	ktime_get_real_ts64(&val1);
 	len = scnprintf(debug->debug_buffer + len, buf_size - len,
@@ -266,6 +267,16 @@ int fnic_get_stats_data(struct stats_debug_info *debug,
 		  (u64)atomic64_read(&stats->io_stats.io_btw_10000_to_30000_msec),
 		  (u64)atomic64_read(&stats->io_stats.io_greater_than_30000_msec));
 
+	len += scnprintf(debug->debug_buffer + len, buf_size - len,
+			"------------------------------------------\n"
+			"\t\tIO Queues and cumulative IOs\n"
+			"------------------------------------------\n");
+
+	for (i = 0; i < FNIC_MQ_MAX_QUEUES; i++) {
+		len += scnprintf(debug->debug_buffer + len, buf_size - len,
+				"Q:%d -> %lld\n", i, (u64)atomic64_read(&stats->io_stats.ios[i]));
+	}
+
 	len += scnprintf(debug->debug_buffer + len, buf_size - len,
 		  "\nCurrent Max IO time : %lld\n",
 		  (u64)atomic64_read(&stats->io_stats.current_max_io_time));
Signature · 1 line
2.31.1
author maintainer
#9 lore raw

PATCH v6 9/13 scsi: fnic: Remove usage of host_lock

In reply to Karan Tilak Kumar

Remove usage of host_lock.
Replace with fnic_lock, where necessary.
fnic does not use host_lock. fnic uses fnic_lock. Use fnic lock to protect fnic members in fnic_queuecommand. Add log messages in error cases.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Squash the following commits into one:
scsi: fnic: Remove usage of host_lock
scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand

---
 drivers/scsi/fnic/fnic_scsi.c | 55 ++++++++++++++++++++---------------
 1 file changed, 31 insertions(+), 24 deletions(-)
drivers/scsi/fnic/fnic_scsi.c +31 -24
index c7f3bfc650d1..e9acb2e7dd2e 100644
@@ -170,17 +170,14 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags,
 			unsigned long clearbits)
 {
 	unsigned long flags = 0;
-	unsigned long host_lock_flags = 0;
 
 	spin_lock_irqsave(&fnic->fnic_lock, flags);
-	spin_lock_irqsave(fnic->lport->host->host_lock, host_lock_flags);
 
 	if (clearbits)
 		fnic->state_flags &= ~st_flags;
 	else
 		fnic->state_flags |= st_flags;
 
-	spin_unlock_irqrestore(fnic->lport->host->host_lock, host_lock_flags);
 	spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 
 	return;
@@ -427,14 +424,27 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 	int io_lock_acquired = 0;
 	struct fc_rport_libfc_priv *rp;
 
-	if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_IO_BLOCKED)))
+	spin_lock_irqsave(&fnic->fnic_lock, flags);
+
+	if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_IO_BLOCKED))) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
+		FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host,
+			"fnic<%d>: %s: %d: fnic IO blocked flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n",
+			fnic->fnic_num, __func__, __LINE__, fnic->state_flags);
 		return SCSI_MLQUEUE_HOST_BUSY;
+	}
 
-	if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_FWRESET)))
+	if (unlikely(fnic_chk_state_flags_locked(fnic, FNIC_FLAGS_FWRESET))) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
+		FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host,
+			"fnic<%d>: %s: %d: fnic flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n",
+			fnic->fnic_num, __func__, __LINE__, fnic->state_flags);
 		return SCSI_MLQUEUE_HOST_BUSY;
+	}
 
 	rport = starget_to_rport(scsi_target(sc->device));
 	if (!rport) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 		FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host,
 				"returning DID_NO_CONNECT for IO as rport is NULL\n");
 		sc->result = DID_NO_CONNECT << 16;
@@ -444,6 +454,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 
 	ret = fc_remote_port_chkready(rport);
 	if (ret) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 		FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host,
 				"rport is not ready\n");
 		atomic64_inc(&fnic_stats->misc_stats.rport_not_ready);
@@ -454,6 +465,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 
 	rp = rport->dd_data;
 	if (!rp || rp->rp_state == RPORT_ST_DELETE) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 		FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host,
 			"rport 0x%x removed, returning DID_NO_CONNECT\n",
 			rport->port_id);
@@ -465,6 +477,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 	}
 
 	if (rp->rp_state != RPORT_ST_READY) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 		FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host,
 			"rport 0x%x in state 0x%x, returning DID_IMM_RETRY\n",
 			rport->port_id, rp->rp_state);
@@ -474,17 +487,17 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 		return 0;
 	}
 
-	if (lp->state != LPORT_ST_READY || !(lp->link_up))
+	if (lp->state != LPORT_ST_READY || !(lp->link_up)) {
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
+		FNIC_SCSI_DBG(KERN_ERR, fnic->lport->host,
+			"fnic<%d>: %s: %d: state not ready: %d/link not up: %d Returning HOST_BUSY\n",
+			fnic->fnic_num, __func__, __LINE__, lp->state, lp->link_up);
 		return SCSI_MLQUEUE_HOST_BUSY;
+	}
 
 	atomic_inc(&fnic->in_flight);
 
-	/*
-	 * Release host lock, use driver resource specific locks from here.
-	 * Don't re-enable interrupts in case they were disabled prior to the
-	 * caller disabling them.
-	 */
-	spin_unlock(lp->host->host_lock);
+	spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 	fnic_priv(sc)->state = FNIC_IOREQ_NOT_INITED;
 	fnic_priv(sc)->flags = FNIC_NO_FLAGS;
 
@@ -569,8 +582,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 			mempool_free(io_req, fnic->io_req_pool);
 		}
 		atomic_dec(&fnic->in_flight);
-		/* acquire host lock before returning to SCSI */
-		spin_lock(lp->host->host_lock);
 		return ret;
 	} else {
 		atomic64_inc(&fnic_stats->io_stats.active_ios);
@@ -598,8 +609,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
 		spin_unlock_irqrestore(io_lock, flags);
 
 	atomic_dec(&fnic->in_flight);
-	/* acquire host lock before returning to SCSI */
-	spin_lock(lp->host->host_lock);
 	return ret;
 }
 
@@ -1493,18 +1502,17 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag,
 					  struct fnic_io_req *io_req)
 {
 	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
-	struct Scsi_Host *host = fnic->lport->host;
 	struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;
 	unsigned long flags;
 
-	spin_lock_irqsave(host->host_lock, flags);
+	spin_lock_irqsave(&fnic->fnic_lock, flags);
 	if (unlikely(fnic_chk_state_flags_locked(fnic,
 						FNIC_FLAGS_IO_BLOCKED))) {
-		spin_unlock_irqrestore(host->host_lock, flags);
+		spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 		return 1;
 	} else
 		atomic_inc(&fnic->in_flight);
-	spin_unlock_irqrestore(host->host_lock, flags);
+	spin_unlock_irqrestore(&fnic->fnic_lock, flags);
 
 	spin_lock_irqsave(&fnic->wq_copy_lock[0], flags);
 
@@ -1939,7 +1947,6 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic,
 				       struct fnic_io_req *io_req)
 {
 	struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];
-	struct Scsi_Host *host = fnic->lport->host;
 	struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;
 	struct scsi_lun fc_lun;
 	int ret = 0;
@@ -1949,14 +1956,14 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic,
 	if (tag == SCSI_NO_TAG)
 		tag = io_req->tag;
 
-	spin_lock_irqsave(host->host_lock, intr_flags);
+	spin_lock_irqsave(&fnic->fnic_lock, intr_flags);
 	if (unlikely(fnic_chk_state_flags_locked(fnic,
 						FNIC_FLAGS_IO_BLOCKED))) {
-		spin_unlock_irqrestore(host->host_lock, intr_flags);
+		spin_unlock_irqrestore(&fnic->fnic_lock, intr_flags);
 		return FAILED;
 	} else
 		atomic_inc(&fnic->in_flight);
-	spin_unlock_irqrestore(host->host_lock, intr_flags);
+	spin_unlock_irqrestore(&fnic->fnic_lock, intr_flags);
 
 	spin_lock_irqsave(&fnic->wq_copy_lock[0], intr_flags);
 
Signature · 1 line
2.31.1
author maintainer
#10 lore raw

PATCH v6 10/13 scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c

In reply to Karan Tilak Kumar

Set map_queues in the fnic_host_template to fnic_mq_map_queues_cpus. Define fnic_mq_map_queues_cpus to set cpu assignment to
fnic queues.
Refactor code in fnic_probe to enable vnic queues before scsi_add_host. Modify notify set to the correct index.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate review comment from Hannes:
Replace cpy_wq_base with copy_wq_base.
Incorporate review comment from John Garry:
Replace code in fnic_mq_map_queues_cpus
with blk_mq_pci_map_queues.
Replace shost_printk logs with FNIC_MAIN_DBG.

---
 drivers/scsi/fnic/fnic.h      |   4 +-
 drivers/scsi/fnic/fnic_main.c | 107 ++++++++++++++++++++++++----------
 2 files changed, 77 insertions(+), 34 deletions(-)
drivers/scsi/fnic/fnic.h +2 -2
index 8b3e4fe1ce92..879e0f5b9b0a 100644
@@ -109,7 +109,7 @@ static inline u64 fnic_flags_and_state(struct scsi_cmnd *cmd)
 #define FNIC_ABT_TERM_DELAY_TIMEOUT  500        /* mSec */
 
 #define FNIC_MAX_FCP_TARGET     256
-
+#define FNIC_PCI_OFFSET		2
 /**
  * state_flags to identify host state along along with fnic's state
  **/
@@ -386,7 +386,7 @@ void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq,
 int fnic_fw_reset_handler(struct fnic *fnic);
 void fnic_terminate_rport_io(struct fc_rport *);
 const char *fnic_state_to_str(unsigned int state);
-
+void fnic_mq_map_queues_cpus(struct Scsi_Host *host);
 void fnic_log_q_error(struct fnic *fnic);
 void fnic_handle_link_event(struct fnic *fnic);
 
drivers/scsi/fnic/fnic_main.c +75 -32

235 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#11 lore raw

PATCH v6 11/13 scsi: fnic: Add support for multiqueue (MQ) in fnic driver

In reply to Karan Tilak Kumar

Implement support for MQ in fnic driver:

The block multiqueue layer issues IO to the fnic driver with an MQ tag. Use the mqtag and derive a tag from it. Derive the hardware queue from the mqtag and use it in all paths. Modify queuecommand to handle mqtag.

Replace wq and cq indices to support MQ. Replace the zeroth queue with a hardware queue. Implement spin locks on a per hardware queue basis. Replace io_lock with per hardware queue spinlock. Implement out of range tag checks.

Allocate an io_req_table to track status of the io_req.

Test the driver by building it, loading it,
and configuring 64 queues in UCSM. Issue IOs using Medusa on multiple fnics. Enable/disable links to exercise the abort and clean up path.

---
Changes between v5 and v6:
Incorporate comments from Martin:
Rebase changes to 6.8/scsi-queue, Resolve merge conflicts.

Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate the following review comments from Hannes:
Replace cpy_wq_base with copy_wq_base. Remove hwq as an argument to fnic_queuecommand_int.
Suppress warning from kernel test robot. Replace new shost_printk comments with FNIC_SCSI_DBG. Replace fnic_queuecommand_int with fnic_queuecommand.

Changes between v1 and v2:
Incorporate the following review comments from Bart:
Remove outdated comment,
Remove unnecessary out of range tag checks, Remove unnecessary local variable,
Modify function name.

---
 drivers/scsi/fnic/fnic.h      |   2 -
 drivers/scsi/fnic/fnic_main.c |   5 -
 drivers/scsi/fnic/fnic_scsi.c | 575 +++++++++++++++++++---------------
 3 files changed, 328 insertions(+), 254 deletions(-)
drivers/scsi/fnic/fnic.h +0 -2
index 879e0f5b9b0a..34de8f0aacb2 100644
@@ -36,7 +36,6 @@
 #define FNIC_MIN_IO_REQ			256 /* Min IO throttle count */
 #define FNIC_MAX_IO_REQ		1024 /* scsi_cmnd tag map entries */
 #define FNIC_DFLT_IO_REQ        256 /* Default scsi_cmnd tag map entries */
-#define	FNIC_IO_LOCKS		64 /* IO locks: power of 2 */
 #define FNIC_DFLT_QUEUE_DEPTH	256
 #define	FNIC_STATS_RATE_LIMIT	4 /* limit rate at which stats are pulled up */
 
@@ -300,7 +299,6 @@ struct fnic {
 	struct fnic_host_tag *tags;
 	mempool_t *io_req_pool;
 	mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES];
-	spinlock_t io_req_lock[FNIC_IO_LOCKS];	/* locks for scsi cmnds */
 
 	unsigned int copy_wq_base;
 	struct work_struct link_work;
drivers/scsi/fnic/fnic_main.c +0 -5
index 7fee8a024edb..5ed1d897311a 100644
@@ -794,11 +794,6 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		fnic->fw_ack_index[i] = -1;
 	}
 
-	for (i = 0; i < FNIC_IO_LOCKS; i++)
-		spin_lock_init(&fnic->io_req_lock[i]);
-
-	spin_lock_init(&fnic->sgreset_lock);
-
 	err = -ENOMEM;
 	fnic->io_req_pool = mempool_create_slab_pool(2, fnic_io_req_cache);
 	if (!fnic->io_req_pool)
drivers/scsi/fnic/fnic_scsi.c +328 -247

1,383 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#12 lore raw

PATCH v6 12/13 scsi: fnic: Improve logs and add support for multiqueue (MQ)

In reply to Karan Tilak Kumar

Improve existing logs by adding fnic number, hardware queue, tag, and mqtag in the prints.
Add logs with the above elements for effective debugging.

---
Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate the following review comment from Hannes:
Use fnic_num as an argument to FNIC_SCSI_DBG. Modify definition of FNIC_SCSI_DBG.
Host number is still an argument since FNIC_SCSI_DBG in turn uses shost_printk.

---
 drivers/scsi/fnic/fnic.h      |   6 +-
 drivers/scsi/fnic/fnic_scsi.c | 288 ++++++++++++++++++----------------
 2 files changed, 154 insertions(+), 140 deletions(-)
drivers/scsi/fnic/fnic.h +4 -2
index 34de8f0aacb2..28f402932b3c 100644
@@ -155,9 +155,11 @@ do {								\
 				"fnic<%d>: %s: %d: " fmt, fnic_num,\
 				__func__, __LINE__, ##args);)
 
-#define FNIC_SCSI_DBG(kern_level, host, fmt, args...)		\
+#define FNIC_SCSI_DBG(kern_level, host, fnic_num, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_SCSI_LOGGING,			\
-			 shost_printk(kern_level, host, fmt, ##args);)
+			 shost_printk(kern_level, host,			\
+				"fnic<%d>: %s: %d: " fmt, fnic_num,\
+				__func__, __LINE__, ##args);)
 
 #define FNIC_ISR_DBG(kern_level, host, fnic_num, fmt, args...)		\
 	FNIC_CHECK_LOGGING(FNIC_ISR_LOGGING,			\
drivers/scsi/fnic/fnic_scsi.c +150 -138

670 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#13 lore raw

PATCH v6 13/13 scsi: fnic: Increment driver version

In reply to Karan Tilak Kumar

Increment driver version for multiqueue(MQ)

---
Changes between v5 and v6:
Incorporate comments from Martin:
Rebase changes to 6.8/scsi-queue,
Resolve merge conflicts.

Changes between v4 and v5:
Incorporate review comments from Martin:
Modify patch commits to include a "---" separator.

Changes between v2 and v3:
Incorporate the following review comments from Hannes:
Create a separate patch to increment driver version. Increment driver version number to 1.7.0.0.

---
 drivers/scsi/fnic/fnic.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/fnic/fnic.h +1 -1
index 28f402932b3c..2074937c05bc 100644
@@ -27,7 +27,7 @@
 
 #define DRV_NAME		"fnic"
 #define DRV_DESCRIPTION		"Cisco FCoE HBA Driver"
-#define DRV_VERSION		"1.6.0.57"
+#define DRV_VERSION		"1.7.0.0"
 #define PFX			DRV_NAME ": "
 #define DFX                     DRV_NAME "%d: "
 
Signature · 1 line
2.31.1
#14 lore raw
In reply to Karan Tilak Kumar

Karan,

Quoted 2 lines

This cover letter describes the feature: add support for multiqueue (MQ) to fnic driver.

Applied to 6.8/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#15 lore raw
In reply to Karan Tilak Kumar
Quoted 10 lines · Karan Tilak Kumar

This cover letter describes the feature: add support for multiqueue (MQ) to fnic driver.

Background: The Virtual Interface Card (VIC) firmware exposes several queues that can be configured for sending IOs and receiving IO responses. Unified Computing System Manager (UCSM) and Intersight Manager (IMM) allows users to configure the number of queues to be used for IOs.

[...]

Applied to 6.8/scsi-queue, thanks!

[01/13] scsi: fnic: Modify definitions to sync with VIC firmware

        https://git.kernel.org/mkp/scsi/c/2cda90e6f649
[02/13] scsi: fnic: Add and use fnic number
        https://git.kernel.org/mkp/scsi/c/ca008aeeb02c
[03/13] scsi: fnic: Add and improve log messages
        https://git.kernel.org/mkp/scsi/c/3df9dd0d51c2
[04/13] scsi: fnic: Rename wq_copy to hw_copy_wq
        https://git.kernel.org/mkp/scsi/c/86b86a7d2fda
[05/13] scsi: fnic: Get copy workqueue count and interrupt mode from config
        https://git.kernel.org/mkp/scsi/c/8c2a6f815a18
[06/13] scsi: fnic: Refactor and redefine fnic.h for multiqueue
        https://git.kernel.org/mkp/scsi/c/554a14826020
[07/13] scsi: fnic: Modify ISRs to support multiqueue(MQ)
        https://git.kernel.org/mkp/scsi/c/8a8449ca5e33
[08/13] scsi: fnic: Define stats to track multiqueue (MQ) IOs
        https://git.kernel.org/mkp/scsi/c/b559b827716c
[09/13] scsi: fnic: Remove usage of host_lock
        https://git.kernel.org/mkp/scsi/c/848d010ab934
[10/13] scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c
        https://git.kernel.org/mkp/scsi/c/52f6e196e52e
[11/13] scsi: fnic: Add support for multiqueue (MQ) in fnic driver
        https://git.kernel.org/mkp/scsi/c/c81df08cd294
[12/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
        https://git.kernel.org/mkp/scsi/c/55cf715244a7
[13/13] scsi: fnic: Increment driver version
        https://git.kernel.org/mkp/scsi/c/53021c192cc5
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering