Introduce support for multiqueue (MQ) in fnic
Subsystems (2)
Karan Tilak Kumar
maintainerv2 cover Introduce support for multiqueue (MQ) in fnic
Hi Martin,
This cover letter describes the feature: add support for multiqueue (MQ) to fnic driver.
Background: The Virtual Interface Card (VIC) firmware exposes several queues that can be configured for sending IOs and receiving IO responses. Unified Computing System Manager (UCSM) and Intersight Manager (IMM) allows users to configure the number of queues to be used for IOs.
The number of IO queues to be used is stored in a configuration file by the VIC firmware. The fNIC driver reads the configuration file and sets the number of queues to be used. Previously, the driver was hard-coded to use only one queue. With this set of changes, the fNIC driver will configure itself to use multiple queues. This feature takes advantage of the block multiqueue layer to parallelize IOs being sent out of the VIC card.
Here's a brief description of some of the salient patches:
- vnic_scsi.h needs to be in sync with VIC firmware to be able to read the number of queues from the firmware config file. A patch has been created for this.
- In an environment with many fnics (like we see in our customer environments), it is hard to distinguish which fnic is printing logs. Therefore, an fnic number has been included in the logs.
- read the number of queues from the firmware config file.
- include definitions in fnic.h to support multiqueue.
- modify the interrupt service routines (ISRs) to read from the correct registers. The numbers that are used here come from discussions with the VIC firmware team.
- track IO statistics for different queues.
- remove usage of host_lock, and only use fnic_lock in the fnic driver.
- use a hardware queue based spinlock to protect io_req.
- replace the hard-coded zeroth queue with a hardware queue number. This presents a bulk of the changes.
- modify the definition of fnic_queuecommand to accept multiqueue tags.
- improve log messages, and indicate fnic number and multiqueue tags for effective debugging.
Even though the patches have been made into a series, some patches are heavier than others.
But, every effort has been made to keep the purpose of each patch as a single-purpose, and to compile cleanly.
This patchset has been tested as a whole. Therefore, the tested-by fields have been added only to the last two patches
in the set. All the individual patches compile cleanly. However, I've refrained from adding tested-by to
most of the patches, so as to not mislead the reviewer/reader.
A brief note on the unit tests:
1. Increase number of queues to 64. Load driver. Run IOs via Medusa. 12+ hour run successful.
2. Configure multipathing, and run link flaps on single link. IOs drop briefly, but pick up as expected.
3. Configure multipathing, and run link flaps on two links, with a 30 second delay in between. IOs drop briefly, but pick up as expected.
Repeat the above tests with single queue. All tests were successful.
Please consider this patch series for the next merge window.
Changes between v1 and v2:
Suppress a warning raised by a kernel test bot, Incorporate the following review comments from Bart:
Remove outdated comment,
Remove unnecessary out of range tag checks, Remove unnecessary local variable,
Modify function name.
Thanks and regards,
Karan
Karan Tilak Kumar (13):
scsi: fnic: Modify definitions to sync with VIC firmware
scsi: fnic: Add and use fnic number
scsi: fnic: Add and improve log messages
scsi: fnic: Rename wq_copy to hw_copy_wq
scsi: fnic: Get copy workqueue count and interrupt mode from config
scsi: fnic: Refactor and redefine fnic.h for multiqueue
scsi: fnic: Modify ISRs to support multiqueue(MQ)
scsi: fnic: Define stats to track multiqueue (MQ) IOs
scsi: fnic: Remove usage of host_lock
scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c
scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand
scsi: fnic: Add support for multiqueue (MQ) in fnic driver
scsi: fnic: Improve logs and add support for multiqueue (MQ)
drivers/scsi/fnic/fnic.h | 42 +- drivers/scsi/fnic/fnic_debugfs.c | 2 +- drivers/scsi/fnic/fnic_fcs.c | 36 +- drivers/scsi/fnic/fnic_io.h | 2 + drivers/scsi/fnic/fnic_isr.c | 166 +++++-- drivers/scsi/fnic/fnic_main.c | 156 ++++-- drivers/scsi/fnic/fnic_res.c | 48 +- drivers/scsi/fnic/fnic_scsi.c | 789 ++++++++++++++++++------------- drivers/scsi/fnic/fnic_stats.h | 3 + drivers/scsi/fnic/fnic_trace.c | 11 + drivers/scsi/fnic/vnic_dev.c | 4 + drivers/scsi/fnic/vnic_scsi.h | 13 +- 12 files changed, 823 insertions(+), 449 deletions(-)
Signature · 1 line
2.31.1
Karan Tilak Kumar
maintainerPATCH v2 1/13 scsi: fnic: Modify definitions to sync with VIC firmware
VIC firmware has updated definitions.
Modify structure and definitions to sync with the latest VIC firmware.
--- drivers/scsi/fnic/vnic_scsi.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/scsi/fnic/vnic_scsi.h +11 -2
@@ -26,7 +26,7 @@#define VNIC_FNIC_RATOV_MAX 255000#define VNIC_FNIC_MAXDATAFIELDSIZE_MIN 256 -#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX 2112 +#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX 2048#define VNIC_FNIC_FLOGI_RETRIES_MIN 0#define VNIC_FNIC_FLOGI_RETRIES_MAX 0xffffffff
@@ -55,7 +55,7 @@#define VNIC_FNIC_PORT_DOWN_IO_RETRIES_MAX 255#define VNIC_FNIC_LUNS_PER_TARGET_MIN 1 -#define VNIC_FNIC_LUNS_PER_TARGET_MAX 1024 +#define VNIC_FNIC_LUNS_PER_TARGET_MAX 4096/* Device-specific region: scsi configuration */ struct vnic_fc_config {
@@ -79,10 +79,19 @@ struct vnic_fc_config {u16 ra_tov;u16 intr_timer;u8 intr_timer_type; +u8 intr_mode; +u8 lun_queue_depth; +u8 io_timeout_retry; + u16 wq_copy_count; };#define VFCF_FCP_SEQ_LVL_ERR 0x1/* Enable FCP-2 Error Recovery */ #define VFCF_PERBI 0x2/* persistent binding info available */ #define VFCF_FIP_CAPABLE 0x4/* firmware can handle FIP */ +#define VFCF_FC_INITIATOR 0x20/* FC Initiator Mode */ +#define VFCF_FC_TARGET 0x40/* FC Target Mode */ +#define VFCF_FC_NVME_INITIATOR 0x80/* FC-NVMe Initiator Mode */ +#define VFCF_FC_NVME_TARGET 0x100/* FC-NVMe Target Mode */ +#endif /* _VNIC_SCSI_H_ */
Signature · 1 line
2.31.1
Re: [PATCH v2 01/13] scsi: fnic: Modify definitions to sync with VIC firmware
Quoted 42 lines · Karan Tilak Kumar
VIC firmware has updated definitions.
Modify structure and definitions to sync with the latest VIC firmware.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/vnic_scsi.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)drivers/scsi/fnic/vnic_scsi.h +6
-2@@ -26,7 +26,7 @@#define VNIC_FNIC_RATOV_MAX 255000#define VNIC_FNIC_MAXDATAFIELDSIZE_MIN 256 -#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX 2112 +#define VNIC_FNIC_MAXDATAFIELDSIZE_MAX 2048#define VNIC_FNIC_FLOGI_RETRIES_MIN 0#define VNIC_FNIC_FLOGI_RETRIES_MAX 0xffffffff@@ -55,7 +55,7 @@#define VNIC_FNIC_PORT_DOWN_IO_RETRIES_MAX 255#define VNIC_FNIC_LUNS_PER_TARGET_MIN 1 -#define VNIC_FNIC_LUNS_PER_TARGET_MAX 1024 +#define VNIC_FNIC_LUNS_PER_TARGET_MAX 4096/* Device-specific region: scsi configuration */ struct vnic_fc_config {@@ -79,10 +79,19 @@ struct vnic_fc_config {u16 ra_tov;u16 intr_timer;u8 intr_timer_type; +u8 intr_mode; +u8 lun_queue_depth; +u8 io_timeout_retry; +u16 wq_copy_count; };
drivers/scsi/fnic/vnic_scsi.h struct vnic_fc_config { · line 97
Quoted 10 lines
drivers/scsi/fnic/vnic_scsi.h +5
-0#define VFCF_FCP_SEQ_LVL_ERR 0x1/* Enable FCP-2 Error Recovery */ #define VFCF_PERBI 0x2/* persistent binding info available */ #define VFCF_FIP_CAPABLE 0x4/* firmware can handle FIP */ +#define VFCF_FC_INITIATOR 0x20/* FC Initiator Mode */ +#define VFCF_FC_TARGET 0x40/* FC Target Mode */ +#define VFCF_FC_NVME_INITIATOR 0x80/* FC-NVMe Initiator Mode */ +#define VFCF_FC_NVME_TARGET 0x100/* FC-NVMe Target Mode */ +#endif /* _VNIC_SCSI_H_ */
I would have thought that you may want to check for the firmware version here; updating some limits with no check for the version will mean the driver might fail on older firmware, no?
But this is arguably not part of this patch, so:
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 01/13] scsi: fnic: Modify definitions to sync with VIC firmware
Quoted 7 lines · Hannes Reinecke
I would have thought that you may want to check for the firmware version here; updating some limits with no check for the version will mean the driver might fail on older firmware, no?
But this is arguably not part of this patch, so:
Reviewed-by Hannes Reinecke <[email protected]>
Thanks for your review and review comments, Hannes.
Support for these limits has been in firmware for a long time. We are adding support in driver only now.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 2/13 scsi: fnic: Add and use fnic number
In reply to Karan Tilak KumarAdd fnic_num in fnic.h to identify fnic in a multi-fnic environment. Increment and set the fnic number during driver load in fnic_probe. Replace the host number with fnic number in debugfs.
--- drivers/scsi/fnic/fnic.h | 1 + drivers/scsi/fnic/fnic_debugfs.c | 2 +- drivers/scsi/fnic/fnic_main.c | 6 +++++- 3 files changed, 7 insertions(+), 2 deletions(-)
drivers/scsi/fnic/fnic.h +1 -0
@@ -216,6 +216,7 @@ struct fnic_event {/* Per-instance private data structure */ struct fnic { +int fnic_num;struct fc_lport *lport;struct fcoe_ctlr ctlr;/* FIP FCoE controller structure */ struct vnic_dev_bar bar0;
drivers/scsi/fnic/fnic_debugfs.c +1 -1
@@ -676,7 +676,7 @@ void fnic_stats_debugfs_init(struct fnic *fnic) {char name[16 ]; -snprintf (name,sizeof (name),"host%d" , fnic->lport->host->host_no); +snprintf (name,sizeof (name),"fnic%d" , fnic->fnic_num); fnic->fnic_stats_debugfs_host =debugfs_create_dir (name, fnic_stats_debugfs_root);
drivers/scsi/fnic/fnic_main.c +5 -1
@@ -84,6 +84,9 @@ static struct libfc_function_template fnic_transport_template = { .exch_mgr_reset = fnic_exch_mgr_reset }; + +atomic_t fnic_num; +static int fnic_slave_alloc (struct scsi_device *sdev) {struct fc_rport *rport =starget_to_rport (scsi_target (sdev));
@@ -587,6 +590,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)int i;unsigned long flags; +atomic_inc (&fnic_num);/* * Allocate SCSI Host and set up association between host, * local port, and fnic
@@ -608,7 +612,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) host->host_no); host->transportt = fnic_fc_transport; - + fnic->fnic_num =atomic_read (&fnic_num);fnic_stats_debugfs_init (fnic);/* Setup PCI resources */
Signature · 1 line
2.31.1
Re: [PATCH v2 02/13] scsi: fnic: Add and use fnic number
Quoted 4 lines · Karan Tilak Kumar
Add
fnic_numinfnic.hto identify fnic in a multi-fnic environment. Increment and set the fnic number during driver load infnic_probe. Replace the host number with fnic number in debugfs.
Quoted 9 lines
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 1 + drivers/scsi/fnic/fnic_debugfs.c | 2 +- drivers/scsi/fnic/fnic_main.c | 6 +++++- 3 files changed, 7 insertions(+), 2 deletions(-)
Why? The scsi host number gives you a perfectly good enumeration. And really all you do is replacing the scsi host number with an internal number, with no change in functionality.
Why?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 02/13] scsi: fnic: Add and use fnic number
Quoted 15 lines · Hannes Reinecke
Quoted 4 lines · Karan Tilak Kumar
Add
fnic_numinfnic.hto identify fnic in a multi-fnic environment. Increment and set the fnic number during driver load infnic_probe. Replace the host number with fnic number in debugfs.Quoted 9 lines
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 1 + drivers/scsi/fnic/fnic_debugfs.c | 2 +- drivers/scsi/fnic/fnic_main.c | 6 +++++- 3 files changed, 7 insertions(+), 2 deletions(-)
Quoted 8 lines
Why? The scsi host number gives you a perfectly good enumeration.
And really all you do is replacing the scsi host number with an internal number, with no change in functionality.Why?
Cheers,
Hannes
Thanks for your review comments, Hannes.
The scsi host numbers are not contiguous. Having contiguous numbers makes it easier to debug. Also, mapping to VIC hardware instances makes it easier to debug.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 3/13 scsi: fnic: Add and improve log messages
In reply to Karan Tilak KumarAdd link related log messages in fnic_fcs.c, Improve log message in fnic_fcs.c,
Add log message in vnic_dev.c.
--- drivers/scsi/fnic/fnic_fcs.c | 36 ++++++++++++++++++++++++------------ drivers/scsi/fnic/vnic_dev.c | 4 ++++ 2 files changed, 28 insertions(+), 12 deletions(-)
drivers/scsi/fnic/fnic_fcs.c +24 -12
@@ -64,8 +64,8 @@ void fnic_handle_link(struct work_struct *work) new_port_speed);if (old_port_speed != new_port_speed)FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, -"Current vnic speed set to : %llu\n" , - new_port_speed); +"fnic<%d>: %s: %d: Current vnic speed set to: %llu\n" , + fnic->fnic_num, __func__, __LINE__, new_port_speed);switch (vnic_dev_port_speed (fnic->vdev)) {case DCEM_PORTSPEED_10G :
@@ -102,6 +102,9 @@ void fnic_handle_link(struct work_struct *work)fnic_fc_trace_set_data (fnic->lport->host->host_no,FNIC_FC_LE ,"Link Status: DOWN->DOWN" ,strlen ("Link Status: DOWN->DOWN" )); +FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: down->down\n" , + fnic->fnic_num, __func__, __LINE__); }else {if (old_link_down_cnt != fnic->link_down_cnt) {/* UP -> DOWN -> UP */
@@ -128,8 +131,9 @@ void fnic_handle_link(struct work_struct *work)fnic_fcoe_send_vlan_req (fnic);return ; } -FNIC_FCS_DBG (KERN_DEBUG , fnic->lport->host, -"link up\n" ); +FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: up->down->up: Link up\n" , + fnic->fnic_num, __func__, __LINE__);fcoe_ctlr_link_up (&fnic->ctlr); }else {/* UP -> UP */
@@ -138,6 +142,9 @@ void fnic_handle_link(struct work_struct *work) fnic->lport->host->host_no,FNIC_FC_LE ,"Link Status: UP_UP" , strlen("Link Status: UP_UP")); +FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: up->up\n" , + fnic->fnic_num, __func__, __LINE__); } } }else if (fnic->link_status) {
@@ -153,7 +160,9 @@ void fnic_handle_link(struct work_struct *work)return ; } -FNIC_FCS_DBG (KERN_DEBUG , fnic->lport->host,"link up\n" ); +FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: down->up: Link up\n" , + fnic->fnic_num, __func__, __LINE__);fnic_fc_trace_set_data (fnic->lport->host->host_no,FNIC_FC_LE ,"Link Status: DOWN_UP" ,strlen ("Link Status: DOWN_UP" ));fcoe_ctlr_link_up (&fnic->ctlr);
@@ -161,7 +170,9 @@ void fnic_handle_link(struct work_struct *work)/* UP -> DOWN */ fnic->lport->host_stats.link_failure_count++;spin_unlock_irqrestore (&fnic->fnic_lock, flags); -FNIC_FCS_DBG (KERN_DEBUG , fnic->lport->host,"link down\n" ); +FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: up->down: Link down\n" , + fnic->fnic_num, __func__, __LINE__); fnic_fc_trace_set_data( fnic->lport->host->host_no, FNIC_FC_LE, "Link Status: UP_DOWN",
@@ -763,8 +774,9 @@ void fnic_set_port_id(struct fc_lport *lport, u32 port_id, struct fc_frame *fp)u8 *mac;int ret; -FNIC_FCS_DBG (KERN_DEBUG , lport->host,"set port_id %x fp %p\n" , - port_id, fp); +FNIC_FCS_DBG (KERN_DEBUG , lport->host, +"fnic<%d>: %s: %d: set port_id 0x%x fp 0x%p\n" , + fnic->fnic_num, __func__, __LINE__, port_id, fp);/* * If we're clearing the FC_ID, change to use the ctl_src_addr.
@@ -790,10 +802,10 @@ void fnic_set_port_id(struct fc_lport *lport, u32 port_id, struct fc_frame *fp)if (fnic->state ==FNIC_IN_ETH_MODE || fnic->state ==FNIC_IN_FC_MODE ) fnic->state =FNIC_IN_ETH_TRANS_FC_MODE ;else { -FNIC_FCS_DBG (KERN_DEBUG , fnic->lport->host, -"Unexpected fnic state %s while" -" processing flogi resp\n" , -fnic_state_to_str (fnic->state)); +FNIC_FCS_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: Unexpected fnic state: %s processing FLOGI response" , + fnic->fnic_num, __func__, __LINE__, +fnic_state_to_str (fnic->state));spin_unlock_irq (&fnic->fnic_lock);return ; }
drivers/scsi/fnic/vnic_dev.c +4 -0
@@ -143,6 +143,10 @@ static int vnic_dev_discover_res(struct vnic_dev *vdev, vdev->res[type].vaddr = (char __iomem *)bar->vaddr + bar_offset; } +pr_info ("res_type_wq: %d res_type_rq: %d res_type_cq: %d res_type_intr_ctrl: %d\n" , + vdev->res[RES_TYPE_WQ ].count, vdev->res[RES_TYPE_RQ ].count, + vdev->res[RES_TYPE_CQ ].count, vdev->res[RES_TYPE_INTR_CTRL ].count); +return 0 ; }
Signature · 1 line
2.31.1
Re: [PATCH v2 03/13] scsi: fnic: Add and improve log messages
Quoted 17 lines · Karan Tilak Kumar
Add link related log messages in
fnic_fcs.c, Improve log message infnic_fcs.c,
Add log message invnic_dev.c.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_fcs.c | 36 ++++++++++++++++++++++++------------ drivers/scsi/fnic/vnic_dev.c | 4 ++++ 2 files changed, 28 insertions(+), 12 deletions(-)drivers/scsi/fnic/fnic_fcs.c +0
-0
drivers/scsi/fnic/fnic_fcs.c void fnic_handle_link(struct work_struct *work) · line 69
Quoted 8 lines
drivers/scsi/fnic/fnic_fcs.c +2
-2@@ -64,8 +64,8 @@ void fnic_handle_link(struct work_struct *work) new_port_speed);if (old_port_speed != new_port_speed)FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, -"Current vnic speed set to : %llu\n" , - new_port_speed); +"fnic<%d>: %s: %d: Current vnic speed set to: %llu\n" , + fnic->fnic_num, __func__, __LINE__, new_port_speed);
Please update FNIC_MAIN_DBG() to use the 'fnic' structure as an argument, then you don't have to prefix all messages with 'fnic<%d>'.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 03/13] scsi: fnic: Add and improve log messages
Quoted 28 lines · Hannes Reinecke
Quoted 16 lines · Karan Tilak Kumar
Add link related log messages in
fnic_fcs.c, Improve log message infnic_fcs.c, Add log message invnic_dev.c.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_fcs.c | 36 ++++++++++++++++++++++++------------ drivers/scsi/fnic/vnic_dev.c | 4 ++++ 2 files changed, 28 insertions(+), 12 deletions(-)(quoted diff) +0
-0
b/drivers/scsi/fnic/fnic_fcs.cindex55632c67a8f2..203ffec625a4100644drivers/scsi/fnic/fnic_fcs.c void fnic_handle_link(struct work_struct *work) · line 69
Quoted 8 lines
drivers/scsi/fnic/fnic_fcs.c +2
-2@@ -64,8 +64,8 @@ void fnic_handle_link(struct work_struct *work) new_port_speed);if (old_port_speed != new_port_speed)FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, -"Current vnic speed set to : %llu\n" , - new_port_speed); +"fnic<%d>: %s: %d: Current vnic speed set to: %llu\n" , + fnic->fnic_num, __func__, __LINE__, new_port_speed);Please update
FNIC_MAIN_DBG()to use the 'fnic' structure as an argument, then you don't have to prefix all messages with 'fnic<%d>'.
Quoted 3 lines
Cheers,
Hannes
Thanks for the review comments, Hannes. I'll make suitable changes in v3.
Please advise whether I'll need to supersede v2 with a "git send-email --in-reply-to" so that your "Reviewed-by" tag for other patches gets preserved. I'm not exactly sure how the process works. The goal is to reduce the number of re-reviews of patches, if possible.
Any pointers with respect to this will help.
Regards,
Karan
Re: [PATCH v2 03/13] scsi: fnic: Add and improve log messages
Quoted 31 lines · Karan Tilak Kumar (kartilak)
Quoted 30 lines · Hannes Reinecke
Quoted 16 lines · Karan Tilak Kumar
Add link related log messages in
fnic_fcs.c, Improve log message infnic_fcs.c, Add log message invnic_dev.c.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_fcs.c | 36 ++++++++++++++++++++++++------------ drivers/scsi/fnic/vnic_dev.c | 4 ++++ 2 files changed, 28 insertions(+), 12 deletions(-)(quoted diff) +0
-0
b/drivers/scsi/fnic/fnic_fcs.cindex55632c67a8f2..203ffec625a4100644drivers/scsi/fnic/fnic_fcs.c void fnic_handle_link(struct work_struct *work) · line 69
Quoted 8 lines
drivers/scsi/fnic/fnic_fcs.c +2
-2@@ -64,8 +64,8 @@ void fnic_handle_link(struct work_struct *work) new_port_speed);if (old_port_speed != new_port_speed)FNIC_MAIN_DBG (KERN_INFO , fnic->lport->host, -"Current vnic speed set to : %llu\n" , - new_port_speed); +"fnic<%d>: %s: %d: Current vnic speed set to: %llu\n" , + fnic->fnic_num, __func__, __LINE__, new_port_speed);Please update
FNIC_MAIN_DBG()to use the 'fnic' structure as an argument, then you don't have to prefix all messages with 'fnic<%d>'.Cheers,
Quoted 8 lines
Quoted 1 line
Hannes
Thanks for the review comments, Hannes. I'll make suitable changes in v3.
Please advise whether I'll need to supersede v2 with a "git send-email --in-reply-to" so that your "Reviewed-by" tag for other patches gets preserved.
I'm not exactly sure how the process works. The goal is to reduce the number of re-reviews of patches, if possible. Any pointers with respect to this will help.
Please add the 'Reviewed-by' tags for the next submission; that will indicated to reviewers which of these patches already have been reviewed and which need further attention.
Hint: there is the 'b4' tool, which will scrape the patches from the mailing list _and_ add the 'Reviewed-by' tags. Maybe give it a go.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 03/13] scsi: fnic: Add and improve log messages
Quoted 9 lines · Hannes Reinecke
Quoted 6 lines
Thanks for the review comments, Hannes. I'll make suitable changes in v3.
Please advise whether I'll need to supersede v2 with a "git send-email
--in-reply-to" so that your "Reviewed-by" tag for other patches gets preserved.
I'm not exactly sure how the process works. The goal is to reduce the number of re-reviews of patches, if possible. Any pointers with respect to this will help.Please add the 'Reviewed-by' tags for the next submission; that will indicated to reviewers which of these patches already have been reviewed and which need further attention.
Hint: there is the 'b4' tool, which will scrape the patches from the mailing list _and_ add the 'Reviewed-by' tags. Maybe give it a go.
Thanks for this suggestion.
I'm using this tool and preparing patches for v3.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 4/13 scsi: fnic: Rename wq_copy to hw_copy_wq
In reply to Karan Tilak KumarRename wq_copy to hw_copy_wq to accurately describe the copy workqueue. This will also help distinguish this data structure from software data structures that can be introduced.
--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_isr.c | 2 +- drivers/scsi/fnic/fnic_main.c | 8 ++++---- drivers/scsi/fnic/fnic_res.c | 6 +++--- drivers/scsi/fnic/fnic_scsi.c | 12 ++++++------ 5 files changed, 15 insertions(+), 15 deletions(-)
drivers/scsi/fnic/fnic.h +1 -1
@@ -301,7 +301,7 @@ struct fnic {/*** FIP related data members -- end ***/ /* copy work queue cache line section */ -____cacheline_aligned struct vnic_wq_copy wq_copy[FNIC_WQ_COPY_MAX ]; +____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX ];/* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX ];
drivers/scsi/fnic/fnic_isr.c +1 -1
@@ -224,7 +224,7 @@ int fnic_set_intr_mode(struct fnic *fnic) {unsigned int n =ARRAY_SIZE (fnic->rq);unsigned int m =ARRAY_SIZE (fnic->wq); -unsigned int o =ARRAY_SIZE (fnic->wq_copy); +unsigned int o =ARRAY_SIZE (fnic->hw_copy_wq);/* * Set interrupt mode (INTx, MSI, MSI-X) depending
drivers/scsi/fnic/fnic_main.c +4 -4
@@ -357,7 +357,7 @@ void fnic_log_q_error(struct fnic *fnic) }for (i =0 ; i < fnic->wq_copy_count; i++) { - error_status =ioread32 (&fnic->wq_copy[i].ctrl->error_status); + error_status =ioread32 (&fnic->hw_copy_wq[i].ctrl->error_status);if (error_status)shost_printk (KERN_ERR , fnic->lport->host, "CWQ[%d] error_status"
@@ -494,7 +494,7 @@ static int fnic_cleanup(struct fnic *fnic)return err; }for (i =0 ; i < fnic->wq_copy_count; i++) { - err =vnic_wq_copy_disable (&fnic->wq_copy[i]); + err =vnic_wq_copy_disable (&fnic->hw_copy_wq[i]);if (err)return err; }
@@ -510,7 +510,7 @@ static int fnic_cleanup(struct fnic *fnic)for (i =0 ; i < fnic->rq_count; i++)vnic_rq_clean (&fnic->rq[i], fnic_free_rq_buf);for (i =0 ; i < fnic->wq_copy_count; i++) -vnic_wq_copy_clean (&fnic->wq_copy[i], +vnic_wq_copy_clean (&fnic->hw_copy_wq[i], fnic_wq_copy_cleanup_handler);for (i =0 ; i < fnic->cq_count; i++)
@@ -901,7 +901,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)for (i =0 ; i < fnic->raw_wq_count; i++)vnic_wq_enable (&fnic->wq[i]);for (i =0 ; i < fnic->wq_copy_count; i++) -vnic_wq_copy_enable (&fnic->wq_copy[i]); +vnic_wq_copy_enable (&fnic->hw_copy_wq[i]);fc_fabric_login (lp);
drivers/scsi/fnic/fnic_res.c +3 -3
@@ -203,7 +203,7 @@ void fnic_free_vnic_resources(struct fnic *fnic)vnic_wq_free (&fnic->wq[i]);for (i =0 ; i < fnic->wq_copy_count; i++) -vnic_wq_copy_free (&fnic->wq_copy[i]); +vnic_wq_copy_free (&fnic->hw_copy_wq[i]);for (i =0 ; i < fnic->rq_count; i++)vnic_rq_free (&fnic->rq[i]);
@@ -250,7 +250,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic)/* Allocate Copy WQs used for SCSI IOs */ for (i =0 ; i < fnic->wq_copy_count; i++) { - err =vnic_wq_copy_alloc (fnic->vdev, &fnic->wq_copy[i], + err =vnic_wq_copy_alloc (fnic->vdev, &fnic->hw_copy_wq[i], (fnic->raw_wq_count + i), fnic->config.wq_copy_desc_count,sizeof (struct fcpio_host_req ));
@@ -357,7 +357,7 @@ int fnic_alloc_vnic_resources(struct fnic *fnic) }for (i =0 ; i < fnic->wq_copy_count; i++) { -vnic_wq_copy_init (&fnic->wq_copy[i], +vnic_wq_copy_init (&fnic->hw_copy_wq[i],0 /* cq_index 0 - always */ , error_interrupt_enable, error_interrupt_offset);
drivers/scsi/fnic/fnic_scsi.c +6 -6
@@ -193,7 +193,7 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags,*/ int fnic_fw_reset_handler (struct fnic *fnic) { -struct vnic_wq_copy *wq = &fnic->wq_copy[0 ]; +struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0 ];int ret =0 ;unsigned long flags;
@@ -246,7 +246,7 @@ int fnic_fw_reset_handler(struct fnic *fnic)*/ int fnic_flogi_reg_handler (struct fnic *fnic,u32 fc_id) { -struct vnic_wq_copy *wq = &fnic->wq_copy[0 ]; +struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0 ];enum fcpio_flogi_reg_format_type format;struct fc_lport *lp = fnic->lport;u8 gw_mac[ETH_ALEN ];
@@ -551,7 +551,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)fnic_priv (sc)->flags |=FNIC_IO_INITIALIZED ;/* create copy wq desc and enqueue it */ - wq = &fnic->wq_copy[0 ]; + wq = &fnic->hw_copy_wq[0 ]; ret =fnic_queue_wq_copy_desc (fnic, wq, io_req, sc, sg_count);if (ret) {/*
@@ -782,7 +782,7 @@ static inline void fnic_fcpio_ack_handler(struct fnic *fnic,u64 *ox_id_tag = (u64 *)(void *)desc;/* mark the ack state */ - wq = &fnic->wq_copy[cq_index - fnic->raw_wq_count - fnic->rq_count]; + wq = &fnic->hw_copy_wq[cq_index - fnic->raw_wq_count - fnic->rq_count];spin_lock_irqsave (&fnic->wq_copy_lock[0 ], flags); fnic->fnic_stats.misc_stats.last_ack_time = jiffies;
@@ -1486,7 +1486,7 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag, u32 task_req, u8 *fc_lun, struct fnic_io_req *io_req) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];struct Scsi_Host *host = fnic->lport->host;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;unsigned long flags;
@@ -1932,7 +1932,7 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic,struct scsi_cmnd *sc, struct fnic_io_req *io_req) { - struct vnic_wq_copy *wq = &fnic->wq_copy[0]; + struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0];struct Scsi_Host *host = fnic->lport->host;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;struct scsi_lun fc_lun;
Signature · 1 line
2.31.1
Re: [PATCH v2 04/13] scsi: fnic: Rename wq_copy to hw_copy_wq
Quoted 8 lines · Karan Tilak Kumar
Rename
wq_copytohw_copy_wqto accurately describe the copy workqueue. This will also help distinguish this data structure from software data structures that can be introduced.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>
Quoted 8 lines
--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_isr.c | 2 +- drivers/scsi/fnic/fnic_main.c | 8 ++++---- drivers/scsi/fnic/fnic_res.c | 6 +++--- drivers/scsi/fnic/fnic_scsi.c | 12 ++++++------ 5 files changed, 15 insertions(+), 15 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 04/13] scsi: fnic: Rename wq_copy to hw_copy_wq
Quoted 17 lines · Hannes Reinecke
Quoted 7 lines · Karan Tilak Kumar
Rename
wq_copytohw_copy_wqto accurately describe the copy workqueue. This will also help distinguish this data structure from software data structures that can be introduced.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>Quoted 8 lines
--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_isr.c | 2 +- drivers/scsi/fnic/fnic_main.c | 8 ++++---- drivers/scsi/fnic/fnic_res.c | 6 +++--- drivers/scsi/fnic/fnic_scsi.c | 12 ++++++------ 5 files changed, 15 insertions(+), 15 deletions(-)
Quoted 5 lines
Reviewed-by Hannes Reinecke <[email protected]>Cheers,
Hannes
Thanks for the review comments, Hannes.
I'll make suitable changes in v3.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 5/13 scsi: fnic: Get copy workqueue count and interrupt mode from config
In reply to Karan Tilak KumarGet the copy workqueue count and interrupt mode from the configuration. The config can be changed via UCSM. Add logs to print the interrupt mode and copy workqueue count. Add logs to print the vNIC resources.
--- drivers/scsi/fnic/fnic_res.c | 42 ++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
drivers/scsi/fnic/fnic_res.c +36 -6
@@ -57,6 +57,8 @@ int fnic_get_vnic_config(struct fnic *fnic)GET_CONFIG (port_down_timeout);GET_CONFIG (port_down_io_retries);GET_CONFIG (luns_per_tgt); +GET_CONFIG (intr_mode); +GET_CONFIG (wq_copy_count); c->wq_enet_desc_count = min_t(u32, VNIC_FNIC_WQ_DESCS_MAX,
@@ -131,6 +133,12 @@ int fnic_get_vnic_config(struct fnic *fnic) c->intr_timer =min_t (u16,VNIC_INTR_TIMER_MAX , c->intr_timer); c->intr_timer_type = c->intr_timer_type; +/* for older firmware, GET_CONFIG will not return anything */ +if (c->wq_copy_count ==0 ) + c->wq_copy_count =1 ; + + c->wq_copy_count =min_t (u16,FNIC_WQ_COPY_MAX , c->wq_copy_count); + shost_printk(KERN_INFO, fnic->lport->host, "vNIC MAC addr %pM " "wq/wq_copy/rq %d/%d/%d\n",
@@ -161,6 +169,10 @@ int fnic_get_vnic_config(struct fnic *fnic)shost_printk (KERN_INFO , fnic->lport->host,"vNIC port dn io retries %d port dn timeout %d\n" , c->port_down_io_retries, c->port_down_timeout); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC wq_copy_count: %d\n" , c->wq_copy_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC intr mode: %d\n" , c->intr_mode);return 0 ; }
@@ -187,12 +199,25 @@ int fnic_set_nic_config(struct fnic *fnic, u8 rss_default_cpu,void fnic_get_res_counts (struct fnic *fnic) { fnic->wq_count =vnic_dev_get_res_count (fnic->vdev,RES_TYPE_WQ ); - fnic->raw_wq_count = fnic->wq_count -1 ; - fnic->wq_copy_count = fnic->wq_count - fnic->raw_wq_count; + fnic->raw_wq_count =1 ; + fnic->wq_copy_count = fnic->config.wq_copy_count; fnic->rq_count =vnic_dev_get_res_count (fnic->vdev,RES_TYPE_RQ ); fnic->cq_count =vnic_dev_get_res_count (fnic->vdev,RES_TYPE_CQ ); fnic->intr_count =vnic_dev_get_res_count (fnic->vdev,RES_TYPE_INTR_CTRL ); + +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources wq_count: %d\n" , fnic->wq_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources raw_wq_count: %d\n" , fnic->raw_wq_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources wq_copy_count: %d\n" , fnic->wq_copy_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources rq_count: %d\n" , fnic->rq_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources cq_count: %d\n" , fnic->cq_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC fw resources intr_count: %d\n" , fnic->intr_count); }void fnic_free_vnic_resources (struct fnic *fnic)
@@ -234,10 +259,15 @@ int fnic_alloc_vnic_resources(struct fnic *fnic) intr_mode ==VNIC_DEV_INTR_MODE_MSIX ? "MSI-X" : "unknown"); -shost_printk (KERN_INFO , fnic->lport->host,"vNIC resources avail: " -"wq %d cp_wq %d raw_wq %d rq %d cq %d intr %d\n" , - fnic->wq_count, fnic->wq_copy_count, fnic->raw_wq_count, - fnic->rq_count, fnic->cq_count, fnic->intr_count); +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC resources avail: wq %d cp_wq %d raw_wq %d rq %d" , + fnic->wq_count, fnic->wq_copy_count, + fnic->raw_wq_count, fnic->rq_count); + +shost_printk (KERN_INFO , fnic->lport->host, +"vNIC resources avail: cq %d intr %d cpy-wq desc count %d\n" , + fnic->cq_count, fnic->intr_count, + fnic->config.wq_copy_desc_count);/* Allocate Raw WQ used for FCS frames */ for (i =0 ; i < fnic->raw_wq_count; i++) {
Signature · 1 line
2.31.1
Re: [PATCH v2 05/13] scsi: fnic: Get copy workqueue count and interrupt mode from config
Quoted 5 lines · Karan Tilak Kumar
Get the copy workqueue count and interrupt mode from the configuration. The config can be changed via UCSM. Add logs to print the interrupt mode and copy workqueue count. Add logs to print the vNIC resources.
Quoted 7 lines
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_res.c | 42 ++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 05/13] scsi: fnic: Get copy workqueue count and interrupt mode from config
Quoted 14 lines · Hannes Reinecke
Quoted 5 lines · Karan Tilak Kumar
Get the copy workqueue count and interrupt mode from the configuration. The config can be changed via UCSM. Add logs to print the interrupt mode and copy workqueue count. Add logs to print the vNIC resources.
Quoted 7 lines
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_res.c | 42 ++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
Quoted 5 lines
Reviewed-by Hannes Reinecke <[email protected]>Cheers,
Hannes
Thanks for your review, Hannes.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 6/13 scsi: fnic: Refactor and redefine fnic.h for multiqueue
In reply to Karan Tilak KumarRefactor and re-define values in fnic.h to implement multiqueue(MQ) functionality.
VIC firmware allows fnic to create up to 64 copy workqueues. Update the copy workqueue max to 64. Modify the interrupt index to be in sync with the firmware to support MQ.
Add irq number to the MSIX entry.
Define a software workqueue table to track the status of io_reqs. Define a base for the copy workqueue.
--- drivers/scsi/fnic/fnic.h | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
drivers/scsi/fnic/fnic.h +22 -8
@@ -163,12 +163,21 @@ do { \ #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...) \ shost_printk(kern_level, host, fmt, ##args) +#define FNIC_WQ_COPY_MAX 64 +#define FNIC_WQ_MAX 1 +#define FNIC_RQ_MAX 1 +#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) +#define FNIC_DFLT_IO_COMPLETIONS 256 + +#define FNIC_MQ_CQ_INDEX 2 +extern const char *fnic_state_str[];enum fnic_intx_intr_index {FNIC_INTX_WQ_RQ_COPYWQ , -FNIC_INTX_ERR , +FNIC_INTX_DUMMY ,FNIC_INTX_NOTIFY , +FNIC_INTX_ERR ,FNIC_INTX_INTR_MAX , };
@@ -176,7 +185,7 @@ enum fnic_msix_intr_index {FNIC_MSIX_RQ ,FNIC_MSIX_WQ ,FNIC_MSIX_WQ_COPY , -FNIC_MSIX_ERR_NOTIFY , +FNIC_MSIX_ERR_NOTIFY =FNIC_MSIX_WQ_COPY +FNIC_WQ_COPY_MAX , FNIC_MSIX_INTR_MAX, };
@@ -185,6 +194,7 @@ struct fnic_msix_entry {char devname[IFNAMSIZ +11 ];irqreturn_t (*isr)(int ,void *);void *devid; +int irq_num; };enum fnic_state {
@@ -194,12 +204,6 @@ enum fnic_state { FNIC_IN_ETH_TRANS_FC_MODE, }; -#define FNIC_WQ_COPY_MAX 1 -#define FNIC_WQ_MAX 1 -#define FNIC_RQ_MAX 1 -#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) -#define FNIC_DFLT_IO_COMPLETIONS 256 -struct mempool ;enum fnic_evt {
@@ -214,6 +218,13 @@ struct fnic_event {enum fnic_evt event; }; +struct fnic_cpy_wq { +unsigned long hw_lock_flags; +u16 active_ioreq_count; +u16 ioreq_table_size; +____cacheline_aligned struct fnic_io_req **io_req_table; +}; +/* Per-instance private data structure */ struct fnic {int fnic_num;
@@ -283,6 +294,7 @@ struct fnic {mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES ];spinlock_t io_req_lock[FNIC_IO_LOCKS ];/* locks for scsi cmnds */ +unsigned int cpy_wq_base;struct work_struct link_work;struct work_struct frame_work;struct sk_buff_head frame_queue;
@@ -302,6 +314,8 @@ struct fnic {/* copy work queue cache line section */ ____cacheline_aligned struct vnic_wq_copy hw_copy_wq[FNIC_WQ_COPY_MAX ]; +____cacheline_aligned struct fnic_cpy_wq sw_copy_wq[FNIC_WQ_COPY_MAX ]; +/* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[FNIC_CQ_MAX ];
Signature · 1 line
2.31.1
Re: [PATCH v2 06/13] scsi: fnic: Refactor and redefine fnic.h for multiqueue
Quoted 90 lines · Karan Tilak Kumar
Refactor and re-define values in
fnic.hto implement multiqueue(MQ) functionality.VIC firmware allows fnic to create up to 64 copy workqueues. Update the copy workqueue max to 64. Modify the interrupt index to be in sync with the firmware to support MQ.
Add irq number to the MSIX entry.
Define a software workqueue table to track the status ofio_reqs. Define a base for the copy workqueue.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)drivers/scsi/fnic/fnic.h +19
-8@@ -163,12 +163,21 @@ do { \ #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...) \ shost_printk(kern_level, host, fmt, ##args) +#define FNIC_WQ_COPY_MAX 64 +#define FNIC_WQ_MAX 1 +#define FNIC_RQ_MAX 1 +#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) +#define FNIC_DFLT_IO_COMPLETIONS 256 + +#define FNIC_MQ_CQ_INDEX 2 +extern const char *fnic_state_str[];enum fnic_intx_intr_index {FNIC_INTX_WQ_RQ_COPYWQ , -FNIC_INTX_ERR , +FNIC_INTX_DUMMY ,FNIC_INTX_NOTIFY , +FNIC_INTX_ERR ,FNIC_INTX_INTR_MAX , };@@ -176,7 +185,7 @@ enum fnic_msix_intr_index {FNIC_MSIX_RQ ,FNIC_MSIX_WQ ,FNIC_MSIX_WQ_COPY , -FNIC_MSIX_ERR_NOTIFY , +FNIC_MSIX_ERR_NOTIFY =FNIC_MSIX_WQ_COPY +FNIC_WQ_COPY_MAX , FNIC_MSIX_INTR_MAX, };@@ -185,6 +194,7 @@ struct fnic_msix_entry {char devname[IFNAMSIZ +11 ];irqreturn_t (*isr)(int ,void *);void *devid; +int irq_num; };enum fnic_state {@@ -194,12 +204,6 @@ enum fnic_state { FNIC_IN_ETH_TRANS_FC_MODE, }; -#define FNIC_WQ_COPY_MAX 1 -#define FNIC_WQ_MAX 1 -#define FNIC_RQ_MAX 1 -#define FNIC_CQ_MAX (FNIC_WQ_COPY_MAX + FNIC_WQ_MAX + FNIC_RQ_MAX) -#define FNIC_DFLT_IO_COMPLETIONS 256 -struct mempool ;enum fnic_evt {@@ -214,6 +218,13 @@ struct fnic_event {enum fnic_evt event; }; +struct fnic_cpy_wq { +unsigned long hw_lock_flags; +u16 active_ioreq_count; +u16 ioreq_table_size; +____cacheline_aligned struct fnic_io_req **io_req_table; +}; +/* Per-instance private data structure */ struct fnic {int fnic_num;
drivers/scsi/fnic/fnic.h struct fnic { · line 297
Quoted 4 lines
drivers/scsi/fnic/fnic.h +1
-0@@ -283,6 +294,7 @@ struct fnic {mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES ];spinlock_t io_req_lock[FNIC_IO_LOCKS ];/* locks for scsi cmnds */ +unsigned int cpy_wq_base;
Please name it 'copy_wq_base'; all other instances in the driver always names the structures 'copy', not 'cpy' too.
You are not really saving anything by omitting a single character.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 06/13] scsi: fnic: Refactor and redefine fnic.h for multiqueue
Quoted 3 lines · Hannes Reinecke
Please name it '
copy_wq_base'; all other instances in the driver always names the structures 'copy', not 'cpy' too. You are not really saving anything by omitting a single character.
Sure, I'll make suitable changes in v3.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 7/13 scsi: fnic: Modify ISRs to support multiqueue(MQ)
In reply to Karan Tilak KumarModify interrupt service routines for INTx, MSI, and MSI-x to support multiqueue.
Modify parameter list of fnic_wq_copy_cmpl_handler to take cq_index.
Modify fnic_cleanup function to use the new function call of fnic_wq_copy_cmpl_handler. Refactor code to set
interrupt mode to MSI-x to a new function.
Add a new stat for intx_dummy.
Changes between v1 and v2:
Suppress warning from kernel test bot.
--- drivers/scsi/fnic/fnic.h | 3 +- drivers/scsi/fnic/fnic_isr.c | 164 ++++++++++++++++++++++++--------- drivers/scsi/fnic/fnic_main.c | 4 +- drivers/scsi/fnic/fnic_scsi.c | 38 +++----- drivers/scsi/fnic/fnic_stats.h | 1 + 5 files changed, 140 insertions(+), 70 deletions(-)
drivers/scsi/fnic/fnic.h +2 -1
@@ -343,6 +343,7 @@ extern const struct attribute_group *fnic_host_groups[];void fnic_clear_intr_mode (struct fnic *fnic);int fnic_set_intr_mode (struct fnic *fnic); +int fnic_set_intr_mode_msix (struct fnic *fnic);void fnic_free_intr (struct fnic *fnic);int fnic_request_intr (struct fnic *fnic);
@@ -369,7 +370,7 @@ void fnic_scsi_cleanup(struct fc_lport *);void fnic_scsi_abort_io (struct fc_lport *);void fnic_empty_scsi_cleanup (struct fc_lport *);void fnic_exch_mgr_reset (struct fc_lport *,u32 ,u32 ); -int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int ); +int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int copy_work_to_do,unsigned int cq_index);int fnic_wq_cmpl_handler (struct fnic *fnic,int );int fnic_flogi_reg_handler (struct fnic *fnic,u32 ); void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq,
drivers/scsi/fnic/fnic_isr.c +120 -44
224 lines not on this page · raw message
drivers/scsi/fnic/fnic_main.c +3 -1
@@ -478,6 +478,7 @@ static int fnic_cleanup(struct fnic *fnic) {unsigned int i;int err; +int raw_wq_rq_counts;vnic_dev_disable (fnic->vdev);for (i =0 ; i < fnic->intr_count; i++)
@@ -497,10 +498,11 @@ static int fnic_cleanup(struct fnic *fnic) err =vnic_wq_copy_disable (&fnic->hw_copy_wq[i]);if (err)return err; + raw_wq_rq_counts = fnic->raw_wq_count + fnic->rq_count; +fnic_wq_copy_cmpl_handler (fnic,-1 , i + raw_wq_rq_counts); }/* Clean up completed IOs and FCS frames */ -fnic_wq_copy_cmpl_handler (fnic, io_completions);fnic_wq_cmpl_handler (fnic,-1 );fnic_rq_cmpl_handler (fnic,-1 );
drivers/scsi/fnic/fnic_scsi.c +14 -24
@@ -1303,10 +1303,8 @@ static int fnic_fcpio_cmpl_handler(struct vnic_dev *vdev,* fnic_wq_copy_cmpl_handler * Routine to process wq copy */ -int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int copy_work_to_do) +int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int copy_work_to_do,unsigned int cq_index) { -unsigned int wq_work_done =0 ; -unsigned int i, cq_index;unsigned int cur_work_done;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;u64 start_jiffies =0 ;
@@ -1314,28 +1312,20 @@ int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do)u64 delta_jiffies =0 ;u64 delta_ms =0 ; -for (i =0 ; i < fnic->wq_copy_count; i++) { - cq_index = i + fnic->raw_wq_count + fnic->rq_count; - - start_jiffies = jiffies; - cur_work_done =vnic_cq_copy_service (&fnic->cq[cq_index], - fnic_fcpio_cmpl_handler, - copy_work_to_do); - end_jiffies = jiffies; - - wq_work_done += cur_work_done; - delta_jiffies = end_jiffies - start_jiffies; -if (delta_jiffies > - (u64 )atomic64_read (&misc_stats->max_isr_jiffies)) { -atomic64_set (&misc_stats->max_isr_jiffies, - delta_jiffies); - delta_ms =jiffies_to_msecs (delta_jiffies); -atomic64_set (&misc_stats->max_isr_time_ms, delta_ms); -atomic64_set (&misc_stats->corr_work_done, - cur_work_done); - } + start_jiffies = jiffies; + cur_work_done =vnic_cq_copy_service (&fnic->cq[cq_index], + fnic_fcpio_cmpl_handler, + copy_work_to_do); + end_jiffies = jiffies; + delta_jiffies = end_jiffies - start_jiffies; +if (delta_jiffies > (u64 )atomic64_read (&misc_stats->max_isr_jiffies)) { +atomic64_set (&misc_stats->max_isr_jiffies, delta_jiffies); + delta_ms =jiffies_to_msecs (delta_jiffies); +atomic64_set (&misc_stats->max_isr_time_ms, delta_ms); +atomic64_set (&misc_stats->corr_work_done, cur_work_done); } -return wq_work_done; + +return cur_work_done; }static bool fnic_cleanup_io_iter (struct scsi_cmnd *sc,void *data)
drivers/scsi/fnic/fnic_stats.h +1 -0
@@ -103,6 +103,7 @@ struct misc_stats {atomic64_t rport_not_ready;atomic64_t frame_errors;atomic64_t current_port_speed; +atomic64_t intx_dummy; };struct fnic_stats {
Signature · 1 line
2.31.1
Re: [PATCH v2 07/13] scsi: fnic: Modify ISRs to support multiqueue(MQ)
Quoted 254 lines · Karan Tilak Kumar
Modify interrupt service routines for INTx, MSI, and MSI-x to support multiqueue.
Modify parameter list offnic_wq_copy_cmpl_handlerto takecq_index.
Modifyfnic_cleanupfunction to use the new function call offnic_wq_copy_cmpl_handler. Refactor code to set
interrupt mode to MSI-x to a new function.
Add a new stat forintx_dummy.Changes between v1 and v2:
Suppress warning from kernel test bot.Reported-by kernel test robot <[email protected]>Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 3 +- drivers/scsi/fnic/fnic_isr.c | 164 ++++++++++++++++++++++++--------- drivers/scsi/fnic/fnic_main.c | 4 +- drivers/scsi/fnic/fnic_scsi.c | 38 +++----- drivers/scsi/fnic/fnic_stats.h | 1 + 5 files changed, 140 insertions(+), 70 deletions(-)drivers/scsi/fnic/fnic.h +2
-1@@ -343,6 +343,7 @@ extern const struct attribute_group *fnic_host_groups[];void fnic_clear_intr_mode (struct fnic *fnic);int fnic_set_intr_mode (struct fnic *fnic); +int fnic_set_intr_mode_msix (struct fnic *fnic);void fnic_free_intr (struct fnic *fnic);int fnic_request_intr (struct fnic *fnic);@@ -369,7 +370,7 @@ void fnic_scsi_cleanup(struct fc_lport *);void fnic_scsi_abort_io (struct fc_lport *);void fnic_empty_scsi_cleanup (struct fc_lport *);void fnic_exch_mgr_reset (struct fc_lport *,u32 ,u32 ); -int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int ); +int fnic_wq_copy_cmpl_handler (struct fnic *fnic,int copy_work_to_do,unsigned int cq_index);int fnic_wq_cmpl_handler (struct fnic *fnic,int );int fnic_flogi_reg_handler (struct fnic *fnic,u32 ); void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq,drivers/scsi/fnic/fnic_isr.c +99
-44@@ -38,8 +38,13 @@ static irqreturn_t fnic_isr_legacy(int irq, void *data)fnic_log_q_error (fnic); } +if (pba & (1 <<FNIC_INTX_DUMMY )) { +atomic64_inc (&fnic->fnic_stats.misc_stats.intx_dummy); +vnic_intr_return_all_credits (&fnic->intr[FNIC_INTX_DUMMY ]); + } +if (pba & (1 <<FNIC_INTX_WQ_RQ_COPYWQ )) { - work_done +=fnic_wq_copy_cmpl_handler (fnic, io_completions); + work_done +=fnic_wq_copy_cmpl_handler (fnic, io_completions,FNIC_MQ_CQ_INDEX ); work_done +=fnic_wq_cmpl_handler (fnic,-1 ); work_done +=fnic_rq_cmpl_handler (fnic,-1 );@@ -60,7 +65,7 @@ static irqreturn_t fnic_isr_msi(int irq, void *data) fnic->fnic_stats.misc_stats.last_isr_time = jiffies;atomic64_inc (&fnic->fnic_stats.misc_stats.isr_count); - work_done +=fnic_wq_copy_cmpl_handler (fnic, io_completions); + work_done +=fnic_wq_copy_cmpl_handler (fnic, io_completions,FNIC_MQ_CQ_INDEX ); work_done +=fnic_wq_cmpl_handler (fnic,-1 ); work_done +=fnic_rq_cmpl_handler (fnic,-1 );@@ -109,12 +114,22 @@ static irqreturn_t fnic_isr_msix_wq_copy(int irq, void *data) {struct fnic *fnic = data;unsigned long wq_copy_work_done =0 ; +int i; fnic->fnic_stats.misc_stats.last_isr_time = jiffies;atomic64_inc (&fnic->fnic_stats.misc_stats.isr_count); - wq_copy_work_done =fnic_wq_copy_cmpl_handler (fnic, io_completions); -vnic_intr_return_credits (&fnic->intr[FNIC_MSIX_WQ_COPY ], + i = irq - fnic->msix[0 ].irq_num; +if (i >= fnic->wq_copy_count + fnic->cpy_wq_base || + i <0 || fnic->msix[i].irq_num != irq) { +for (i = fnic->cpy_wq_base; i < fnic->wq_copy_count + fnic->cpy_wq_base ; i++) { +if (fnic->msix[i].irq_num == irq) +break ; + } + } + + wq_copy_work_done =fnic_wq_copy_cmpl_handler (fnic, io_completions, i); +vnic_intr_return_credits (&fnic->intr[i], wq_copy_work_done,1 /* unmask intr */ ,1 /* reset intr timer */ );@@ -128,7 +143,7 @@ static irqreturn_t fnic_isr_msix_err_notify(int irq, void *data) fnic->fnic_stats.misc_stats.last_isr_time = jiffies;atomic64_inc (&fnic->fnic_stats.misc_stats.isr_count); -vnic_intr_return_all_credits (&fnic->intr[FNIC_MSIX_ERR_NOTIFY ]); +vnic_intr_return_all_credits (&fnic->intr[fnic->err_intr_offset]);fnic_log_q_error (fnic);fnic_handle_link_event (fnic);@@ -186,26 +201,30 @@ int fnic_request_intr(struct fnic *fnic) fnic->msix[FNIC_MSIX_WQ ].isr = fnic_isr_msix_wq; fnic->msix[FNIC_MSIX_WQ ].devid = fnic; -sprintf (fnic->msix[FNIC_MSIX_WQ_COPY ].devname, -"%.11s-scsi-wq" , fnic->name); - fnic->msix[FNIC_MSIX_WQ_COPY ].isr = fnic_isr_msix_wq_copy; - fnic->msix[FNIC_MSIX_WQ_COPY ].devid = fnic; +for (i = fnic->cpy_wq_base; i < fnic->wq_copy_count + fnic->cpy_wq_base; i++) { +sprintf (fnic->msix[i].devname, +"%.11s-scsi-wq-%d" , fnic->name, i-FNIC_MSIX_WQ_COPY ); + fnic->msix[i].isr = fnic_isr_msix_wq_copy; + fnic->msix[i].devid = fnic; + } -sprintf (fnic->msix[FNIC_MSIX_ERR_NOTIFY ].devname, +sprintf (fnic->msix[fnic->err_intr_offset].devname,"%.11s-err-notify" , fnic->name); - fnic->msix[FNIC_MSIX_ERR_NOTIFY ].isr = + fnic->msix[fnic->err_intr_offset].isr = fnic_isr_msix_err_notify; - fnic->msix[FNIC_MSIX_ERR_NOTIFY ].devid = fnic; + fnic->msix[fnic->err_intr_offset].devid = fnic; + +for (i =0 ; i < fnic->intr_count; i++) { + fnic->msix[i].irq_num =pci_irq_vector (fnic->pdev, i); -for (i =0 ; i <ARRAY_SIZE (fnic->msix); i++) { - err =request_irq (pci_irq_vector (fnic->pdev, i), - fnic->msix[i].isr,0 , - fnic->msix[i].devname, - fnic->msix[i].devid); + err =request_irq (fnic->msix[i].irq_num, + fnic->msix[i].isr,0 , + fnic->msix[i].devname, + fnic->msix[i].devid);if (err) { -shost_printk (KERN_ERR , fnic->lport->host, -"MSIX: request_irq" -" failed %d\n" , err); +FNIC_ISR_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d request_irq failed with error: %d\n" , + fnic->fnic_num, __func__, __LINE__, err);fnic_free_intr (fnic);break ; }@@ -220,44 +239,101 @@ int fnic_request_intr(struct fnic *fnic)return err; } -int fnic_set_intr_mode (struct fnic *fnic) +int fnic_set_intr_mode_msix (struct fnic *fnic) {unsigned int n =ARRAY_SIZE (fnic->rq);unsigned int m =ARRAY_SIZE (fnic->wq);unsigned int o =ARRAY_SIZE (fnic->hw_copy_wq); +unsigned int min_irqs = n + m +1 +1 ;/*rq, raw wq, wq, err*/ /* -* Set interrupt mode (INTx, MSI, MSI-X) depending -* system capabilities. -* -* Try MSI-X first -* * We need n RQs, m WQs, o Copy WQs, n+m+o CQs, and n+m+o+1 INTRs * (last INTR is used for WQ/RQ errors and notification area) */ -if (fnic->rq_count >= n && - fnic->raw_wq_count >= m && - fnic->wq_copy_count >= o && - fnic->cq_count >= n + m + o) { -int vecs = n + m + o +1 ; - -if (pci_alloc_irq_vectors (fnic->pdev, vecs, vecs, -PCI_IRQ_MSIX ) == vecs) { +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: rq-array size: %d wq-array size: %d copy-wq array size: %d\n" , + fnic->fnic_num, __func__, __LINE__, n, m, o); +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: rq_count: %d raw_wq_count: %d wq_copy_count: %d cq_count: %d\n" , + fnic->fnic_num, __func__, __LINE__, fnic->rq_count, fnic->raw_wq_count, + fnic->wq_copy_count, fnic->cq_count); + +if (fnic->rq_count <= n && fnic->raw_wq_count <= m && + fnic->wq_copy_count <= o) { +int vec_count =0 ; +int vecs = fnic->rq_count + fnic->raw_wq_count + fnic->wq_copy_count +1 ; + + vec_count =pci_alloc_irq_vectors (fnic->pdev, min_irqs, vecs, +PCI_IRQ_MSIX |PCI_IRQ_AFFINITY ); +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: allocated %d MSI-X vectors\n" , + fnic->fnic_num, __func__, __LINE__, vec_count); + +if (vec_count >0 ) { +if (vec_count < vecs) { +FNIC_ISR_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: interrupts number mismatch: vec_count: %d vecs: %d\n" , + fnic->fnic_num, __func__, __LINE__, vec_count, vecs); +if (vec_count < min_irqs) { +FNIC_ISR_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: no interrupts for copy wq\n" , + fnic->fnic_num, __func__, __LINE__); +return 1 ; + } + } + fnic->rq_count = n; fnic->raw_wq_count = m; - fnic->wq_copy_count = o; - fnic->wq_count = m + o; - fnic->cq_count = n + m + o; - fnic->intr_count = vecs; - fnic->err_intr_offset =FNIC_MSIX_ERR_NOTIFY ; - -FNIC_ISR_DBG (KERN_DEBUG , fnic->lport->host, -"Using MSI-X Interrupts\n" ); -vnic_dev_set_intr_mode (fnic->vdev, -VNIC_DEV_INTR_MODE_MSIX ); + fnic->cpy_wq_base = fnic->rq_count + fnic->raw_wq_count; + fnic->wq_copy_count = vec_count - n - m -1 ; + fnic->wq_count = fnic->raw_wq_count + fnic->wq_copy_count; +if (fnic->cq_count != vec_count -1 ) { +FNIC_ISR_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: CQ count: %d does not match MSI-X vector count: %d\n" , + fnic->fnic_num, __func__, __LINE__, fnic->cq_count, vec_count); + fnic->cq_count = vec_count -1 ; + } + fnic->intr_count = vec_count; + fnic->err_intr_offset = fnic->rq_count + fnic->wq_count; + +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: rq_count: %d raw_wq_count: %d cpy_wq_base: %d\n" , + fnic->fnic_num, __func__, __LINE__, fnic->rq_count, + fnic->raw_wq_count, fnic->cpy_wq_base); + +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: wq_copy_count: %d wq_count: %d cq_count: %d\n" , + fnic->fnic_num, __func__, __LINE__, fnic->wq_copy_count, + fnic->wq_count, fnic->cq_count); + +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: intr_count: %d err_intr_offset: %u" , + fnic->fnic_num, __func__, __LINE__, fnic->intr_count, + fnic->err_intr_offset);
drivers/scsi/fnic/fnic_isr.c int fnic_request_intr(struct fnic *fnic) · line 322
Quoted 10 lines
drivers/scsi/fnic/fnic_isr.c +7
-0+ +vnic_dev_set_intr_mode (fnic->vdev,VNIC_DEV_INTR_MODE_MSIX ); +FNIC_ISR_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: fnic using MSI-X\n" , fnic->fnic_num, + __func__, __LINE__);return 0 ; } } +return 1 ; +}//fnic_set_intr_mode_msix
Please avoid C99 style comments.
[ .. ]
Otherwise:
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 07/13] scsi: fnic: Modify ISRs to support multiqueue(MQ)
Quoted 2 lines · Hannes Reinecke
Please avoid C99 style comments.
Sure, I'll make suitable changes in v3.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 8/13 scsi: fnic: Define stats to track multiqueue (MQ) IOs
In reply to Karan Tilak KumarDefine an array to track IOs for the different queues, print the IO stats in fnic get stats data.
--- drivers/scsi/fnic/fnic_stats.h | 2 ++ drivers/scsi/fnic/fnic_trace.c | 11 +++++++++++ 2 files changed, 13 insertions(+)
drivers/scsi/fnic/fnic_stats.h +2 -0
@@ -2,6 +2,7 @@/* Copyright 2013 Cisco Systems, Inc. All rights reserved. */ #ifndef _FNIC_STATS_H_ #define _FNIC_STATS_H_ +#define FNIC_MQ_MAX_QUEUES 64 struct stats_timestamps { struct timespec64 last_reset_time;
@@ -26,6 +27,7 @@ struct io_path_stats {atomic64_t io_btw_10000_to_30000_msec;atomic64_t io_greater_than_30000_msec;atomic64_t current_max_io_time; +atomic64_t ios[FNIC_MQ_MAX_QUEUES ]; };struct abort_stats {
drivers/scsi/fnic/fnic_trace.c +11 -0
@@ -204,6 +204,7 @@ int fnic_get_stats_data(struct stats_debug_info *debug,int len =0 ;int buf_size = debug->buf_size;struct timespec64 val1, val2; +int i =0 ;ktime_get_real_ts64 (&val1); len = scnprintf(debug->debug_buffer + len, buf_size - len,
@@ -266,6 +267,16 @@ int fnic_get_stats_data(struct stats_debug_info *debug, (u64 )atomic64_read (&stats->io_stats.io_btw_10000_to_30000_msec), (u64)atomic64_read(&stats->io_stats.io_greater_than_30000_msec)); + len +=scnprintf (debug->debug_buffer + len, buf_size - len, +"------------------------------------------\n" +"\t\tIO Queues and cumulative IOs\n" +"------------------------------------------\n" ); + +for (i =0 ; i <FNIC_MQ_MAX_QUEUES ; i++) { + len +=scnprintf (debug->debug_buffer + len, buf_size - len, +"Q:%d -> %lld\n" , i, (u64 )atomic64_read (&stats->io_stats.ios[i])); + } + len +=scnprintf (debug->debug_buffer + len, buf_size - len,"\nCurrent Max IO time : %lld\n" , (u64 )atomic64_read (&stats->io_stats.current_max_io_time));
Signature · 1 line
2.31.1
Re: [PATCH v2 08/13] scsi: fnic: Define stats to track multiqueue (MQ) IOs
Quoted 50 lines · Karan Tilak Kumar
Define an array to track IOs for the different queues, print the IO stats in fnic get stats data.
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_stats.h | 2 ++ drivers/scsi/fnic/fnic_trace.c | 11 +++++++++++ 2 files changed, 13 insertions(+)drivers/scsi/fnic/fnic_stats.h +2
-0@@ -2,6 +2,7 @@/* Copyright 2013 Cisco Systems, Inc. All rights reserved. */ #ifndef _FNIC_STATS_H_ #define _FNIC_STATS_H_ +#define FNIC_MQ_MAX_QUEUES 64 struct stats_timestamps { struct timespec64 last_reset_time;@@ -26,6 +27,7 @@ struct io_path_stats {atomic64_t io_btw_10000_to_30000_msec;atomic64_t io_greater_than_30000_msec;atomic64_t current_max_io_time; +atomic64_t ios[FNIC_MQ_MAX_QUEUES ]; };struct abort_stats {drivers/scsi/fnic/fnic_trace.c +4
-0@@ -204,6 +204,7 @@ int fnic_get_stats_data(struct stats_debug_info *debug,int len =0 ;int buf_size = debug->buf_size;struct timespec64 val1, val2; +int i =0 ;ktime_get_real_ts64 (&val1); len = scnprintf(debug->debug_buffer + len, buf_size - len,@@ -266,6 +267,16 @@ int fnic_get_stats_data(struct stats_debug_info *debug, (u64 )atomic64_read (&stats->io_stats.io_btw_10000_to_30000_msec), (u64)atomic64_read(&stats->io_stats.io_greater_than_30000_msec)); + len +=scnprintf (debug->debug_buffer + len, buf_size - len, +"------------------------------------------\n" + "\t\tIO Queues and cumulative IOs\n"
drivers/scsi/fnic/fnic_trace.c int fnic_get_stats_data(struct stats_debug_info *debug, · line 282
Quoted 10 lines
drivers/scsi/fnic/fnic_trace.c +7
-0+ "------------------------------------------\n"); + +for (i =0 ; i <FNIC_MQ_MAX_QUEUES ; i++) { + len +=scnprintf (debug->debug_buffer + len, buf_size - len, +"Q:%d -> %lld\n" , i, (u64 )atomic64_read (&stats->io_stats.ios[i])); + } + len +=scnprintf (debug->debug_buffer + len, buf_size - len,"\nCurrent Max IO time : %lld\n" , (u64 )atomic64_read (&stats->io_stats.current_max_io_time));
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar
maintainerPATCH v2 9/13 scsi: fnic: Remove usage of host_lock
In reply to Karan Tilak KumarRemove usage of host_lock.
Replace with fnic_lock, where necessary.
--- drivers/scsi/fnic/fnic_scsi.c | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-)
drivers/scsi/fnic/fnic_scsi.c +6 -21
@@ -170,17 +170,14 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags, unsigned long clearbits) {unsigned long flags =0 ; -unsigned long host_lock_flags =0 ;spin_lock_irqsave (&fnic->fnic_lock, flags); -spin_lock_irqsave (fnic->lport->host->host_lock, host_lock_flags);if (clearbits) fnic->state_flags &= ~st_flags;else fnic->state_flags |= st_flags; -spin_unlock_irqrestore (fnic->lport->host->host_lock, host_lock_flags);spin_unlock_irqrestore (&fnic->fnic_lock, flags);return ;
@@ -479,12 +476,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)atomic_inc (&fnic->in_flight); -/* -* Release host lock, use driver resource specific locks from here. -* Don't re-enable interrupts in case they were disabled prior to the -* caller disabling them. -*/ -spin_unlock (lp->host->host_lock);fnic_priv (sc)->state =FNIC_IOREQ_NOT_INITED ;fnic_priv (sc)->flags =FNIC_NO_FLAGS ;
@@ -569,8 +560,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)mempool_free (io_req, fnic->io_req_pool); }atomic_dec (&fnic->in_flight); -/* acquire host lock before returning to SCSI */ -spin_lock (lp->host->host_lock);return ret; }else {atomic64_inc (&fnic_stats->io_stats.active_ios);
@@ -598,8 +587,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)spin_unlock_irqrestore (io_lock, flags);atomic_dec (&fnic->in_flight); -/* acquire host lock before returning to SCSI */ -spin_lock (lp->host->host_lock);return ret; }
@@ -1477,18 +1464,17 @@ static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag, struct fnic_io_req *io_req) {struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0 ]; -struct Scsi_Host *host = fnic->lport->host;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;unsigned long flags; -spin_lock_irqsave (host->host_lock, flags); +spin_lock_irqsave (&fnic->fnic_lock, flags);if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_IO_BLOCKED ))) { -spin_unlock_irqrestore (host->host_lock, flags); +spin_unlock_irqrestore (&fnic->fnic_lock, flags);return 1 ; }else atomic_inc (&fnic->in_flight); -spin_unlock_irqrestore (host->host_lock, flags); +spin_unlock_irqrestore (&fnic->fnic_lock, flags);spin_lock_irqsave (&fnic->wq_copy_lock[0 ], flags);
@@ -1923,20 +1909,19 @@ static inline int fnic_queue_dr_io_req(struct fnic *fnic, struct fnic_io_req *io_req) {struct vnic_wq_copy *wq = &fnic->hw_copy_wq[0 ]; -struct Scsi_Host *host = fnic->lport->host;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;struct scsi_lun fc_lun;int ret =0 ;unsigned long intr_flags; -spin_lock_irqsave (host->host_lock, intr_flags); +spin_lock_irqsave (&fnic->fnic_lock, intr_flags);if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_IO_BLOCKED ))) { -spin_unlock_irqrestore (host->host_lock, intr_flags); +spin_unlock_irqrestore (&fnic->fnic_lock, intr_flags);return FAILED ; }else atomic_inc (&fnic->in_flight); -spin_unlock_irqrestore (host->host_lock, intr_flags); +spin_unlock_irqrestore (&fnic->fnic_lock, intr_flags);spin_lock_irqsave (&fnic->wq_copy_lock[0 ], intr_flags);
Signature · 1 line
2.31.1
Re: [PATCH v2 09/13] scsi: fnic: Remove usage of host_lock
Quoted 54 lines · Karan Tilak Kumar
Remove usage of
host_lock.
Replace withfnic_lock, where necessary.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_scsi.c | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-)drivers/scsi/fnic/fnic_scsi.c +0
-11@@ -170,17 +170,14 @@ __fnic_set_state_flags(struct fnic *fnic, unsigned long st_flags, unsigned long clearbits) {unsigned long flags =0 ; -unsigned long host_lock_flags =0 ;spin_lock_irqsave (&fnic->fnic_lock, flags); -spin_lock_irqsave (fnic->lport->host->host_lock, host_lock_flags);if (clearbits) fnic->state_flags &= ~st_flags;else fnic->state_flags |= st_flags; -spin_unlock_irqrestore (fnic->lport->host->host_lock, host_lock_flags);spin_unlock_irqrestore (&fnic->fnic_lock, flags); > return;@@ -479,12 +476,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)atomic_inc (&fnic->in_flight); -/* -* Release host lock, use driver resource specific locks from here. -* Don't re-enable interrupts in case they were disabled prior to the -* caller disabling them. -*/ -spin_unlock (lp->host->host_lock);fnic_priv (sc)->state =FNIC_IOREQ_NOT_INITED ;fnic_priv (sc)->flags =FNIC_NO_FLAGS ;@@ -569,8 +560,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)mempool_free (io_req, fnic->io_req_pool); }atomic_dec (&fnic->in_flight); -/* acquire host lock before returning to SCSI */ -spin_lock (lp->host->host_lock);return ret; }else {atomic64_inc (&fnic_stats->io_stats.active_ios);
drivers/scsi/fnic/fnic_scsi.c static int fnic_queuecommand_lck(struct scsi_cmnd *sc) · line 592
Quoted 8 lines
drivers/scsi/fnic/fnic_scsi.c +0
-2@@ -598,8 +587,6 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)spin_unlock_irqrestore (io_lock, flags);atomic_dec (&fnic->in_flight); -/* acquire host lock before returning to SCSI */ -spin_lock (lp->host->host_lock);return ret; }
If you remove the need for the host_lock during queuecommand() in your driver please rename 'fnic_queuecommand_lck()' to 'fnic_queuecommand()' and remove the line
DEF_SCSI_QCMD(fnic_queuecommand)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 09/13] scsi: fnic: Remove usage of host_lock
Quoted 4 lines · Hannes Reinecke
If you remove the need for the
host_lockduringqueuecommand()in your driver please rename 'fnic_queuecommand_lck()' to 'fnic_queuecommand()' and remove the line
DEF_SCSI_QCMD(fnic_queuecommand)
This change has been made in patch 12/13.
In v3, I've merged patch 9(scsi: fnic: Remove usage of host_lock) with patch 11 (scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand).
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 10/13 scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c
In reply to Karan Tilak KumarSet map_queues in the fnic_host_template to fnic_mq_map_queues_cpus. Define fnic_mq_map_queues_cpus to set cpu assignment to
fnic queues.
Refactor code in fnic_probe to enable vnic queues before scsi_add_host. Modify notify set to the correct index.
--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_main.c | 137 ++++++++++++++++++++++++++-------- 2 files changed, 105 insertions(+), 34 deletions(-)
drivers/scsi/fnic/fnic.h +1 -1
@@ -378,7 +378,7 @@ void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq,int fnic_fw_reset_handler (struct fnic *fnic);void fnic_terminate_rport_io (struct fc_rport *);const char *fnic_state_to_str (unsigned int state); - +void fnic_mq_map_queues_cpus (struct Scsi_Host *host);void fnic_log_q_error (struct fnic *fnic);void fnic_handle_link_event (struct fnic *fnic);
drivers/scsi/fnic/fnic_main.c +104 -33
264 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH v2 10/13] scsi: fnic: Add support for multiqueue (MQ) in fnic_main.c
Quoted 6 lines · Karan Tilak Kumar
Set
map_queuesin thefnic_host_templatetofnic_mq_map_queues_cpus. Definefnic_mq_map_queues_cpusto set cpu assignment to
fnic queues.
Refactor code infnic_probeto enable vnic queues beforescsi_add_host. Modify notify set to the correct index.
Quoted 8 lines
Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_main.c | 137 ++++++++++++++++++++++++++-------- 2 files changed, 105 insertions(+), 34 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar
maintainerPATCH v2 11/13 scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand
In reply to Karan Tilak Kumarfnic does not use host_lock. fnic uses fnic_lock. Use fnic lock to protect fnic members in fnic_queuecommand. Add log messages in error cases.
--- drivers/scsi/fnic/fnic_scsi.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
drivers/scsi/fnic/fnic_scsi.c +25 -3
@@ -424,14 +424,27 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)int io_lock_acquired =0 ;struct fc_rport_libfc_priv *rp; -if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_IO_BLOCKED ))) +spin_lock_irqsave (&fnic->fnic_lock, flags); + +if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_IO_BLOCKED ))) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: fnic IO blocked flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n" , + fnic->fnic_num, __func__, __LINE__, fnic->state_flags);return SCSI_MLQUEUE_HOST_BUSY ; + } -if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_FWRESET ))) +if (unlikely (fnic_chk_state_flags_locked (fnic,FNIC_FLAGS_FWRESET ))) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: fnic flags: 0x%lx. Returning SCSI_MLQUEUE_HOST_BUSY\n" , + fnic->fnic_num, __func__, __LINE__, fnic->state_flags);return SCSI_MLQUEUE_HOST_BUSY ; + } rport =starget_to_rport (scsi_target (sc->device));if (!rport) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags);FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host,"returning DID_NO_CONNECT for IO as rport is NULL\n" ); sc->result =DID_NO_CONNECT <<16 ;
@@ -441,6 +454,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) ret =fc_remote_port_chkready (rport);if (ret) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags);FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host,"rport is not ready\n" );atomic64_inc (&fnic_stats->misc_stats.rport_not_ready);
@@ -451,6 +465,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) rp = rport->dd_data;if (!rp || rp->rp_state ==RPORT_ST_DELETE ) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags);FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host,"rport 0x%x removed, returning DID_NO_CONNECT\n" , rport->port_id);
@@ -462,6 +477,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) }if (rp->rp_state !=RPORT_ST_READY ) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags);FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host,"rport 0x%x in state 0x%x, returning DID_IMM_RETRY\n" , rport->port_id, rp->rp_state);
@@ -471,11 +487,17 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)return 0 ; } -if (lp->state !=LPORT_ST_READY || !(lp->link_up)) +if (lp->state !=LPORT_ST_READY || !(lp->link_up)) { +spin_unlock_irqrestore (&fnic->fnic_lock, flags); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: state not ready: %d/link not up: %d Returning HOST_BUSY\n" , + fnic->fnic_num, __func__, __LINE__, lp->state, lp->link_up);return SCSI_MLQUEUE_HOST_BUSY ; + }atomic_inc (&fnic->in_flight); +spin_unlock_irqrestore (&fnic->fnic_lock, flags);fnic_priv (sc)->state =FNIC_IOREQ_NOT_INITED ;fnic_priv (sc)->flags =FNIC_NO_FLAGS ;
Signature · 1 line
2.31.1
Re: [PATCH v2 11/13] scsi: fnic: Use fnic_lock to protect fnic structures in queuecommand
Quoted 11 lines · Karan Tilak Kumar
fnic does not use
host_lock. fnic usesfnic_lock. Use fnic lock to protect fnic members infnic_queuecommand. Add log messages in error cases.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic_scsi.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
Please merge this patch with the previous one which removes the use of 'host_lock', seeing that you need a lock after all.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar
maintainerPATCH v2 12/13 scsi: fnic: Add support for multiqueue (MQ) in fnic driver
In reply to Karan Tilak KumarImplement support for MQ in fnic driver:
The block multiqueue layer issues IO to the fnic driver with an MQ tag. Use the mqtag and derive a tag from it. Derive the hardware queue from the mqtag and use it in all paths. Modify queuecommand to handle mqtag.
Replace wq and cq indices to support MQ. Replace the zeroth queue with a hardware queue. Implement spin locks on a per hardware queue basis. Replace io_lock with per hardware queue spinlock. Implement out of range tag checks.
Allocate an io_req_table to track status of the io_req.
Test the driver by building it, loading it,
and configuring 64 queues in UCSM. Issue IOs using Medusa on multiple fnics. Enable/disable links to exercise the abort and clean up path.
Changes between v1 and v2:
Incorporate the following review comments from Bart:
Remove outdated comment,
Remove unnecessary out of range tag checks, Remove unnecessary local variable,
Modify function name.
--- drivers/scsi/fnic/fnic.h | 2 - drivers/scsi/fnic/fnic_io.h | 2 + drivers/scsi/fnic/fnic_main.c | 3 - drivers/scsi/fnic/fnic_scsi.c | 571 ++++++++++++++++++++-------------- 4 files changed, 346 insertions(+), 232 deletions(-)
drivers/scsi/fnic/fnic.h +0 -2
@@ -36,7 +36,6 @@#define FNIC_MIN_IO_REQ 256/* Min IO throttle count */ #define FNIC_MAX_IO_REQ 1024/* scsi_cmnd tag map entries */ #define FNIC_DFLT_IO_REQ 256/* Default scsi_cmnd tag map entries */ -#define FNIC_IO_LOCKS 64/* IO locks: power of 2 */ #define FNIC_DFLT_QUEUE_DEPTH 256#define FNIC_STATS_RATE_LIMIT 4/* limit rate at which stats are pulled up */
@@ -292,7 +291,6 @@ struct fnic {struct fnic_host_tag *tags;mempool_t *io_req_pool;mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES ]; -spinlock_t io_req_lock[FNIC_IO_LOCKS ];/* locks for scsi cmnds */ unsigned int cpy_wq_base;struct work_struct link_work;
drivers/scsi/fnic/fnic_io.h +2 -0
@@ -52,6 +52,8 @@ struct fnic_io_req {unsigned long start_time;/* in jiffies */ struct completion *abts_done;/* completion for abts */ struct completion *dr_done;/* completion for device reset */ +unsigned int tag; +struct scsi_cmnd *sc;/* midlayer's cmd pointer */ };enum fnic_port_speeds {
drivers/scsi/fnic/fnic_main.c +0 -3
@@ -816,9 +816,6 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) fnic->fw_ack_index[i] =-1 ; } -for (i =0 ; i <FNIC_IO_LOCKS ; i++) -spin_lock_init (&fnic->io_req_lock[i]); - err = -ENOMEM ; fnic->io_req_pool =mempool_create_slab_pool (2 , fnic_io_req_cache);if (!fnic->io_req_pool)
drivers/scsi/fnic/fnic_scsi.c +344 -227
1,373 lines not on this page · raw message
Signature · 1 line
2.31.1
kernel test robot
build botRe: [PATCH v2 12/13] scsi: fnic: Add support for multiqueue (MQ) in fnic driver
kernel test robot noticed the following build warnings:
Hi Karan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on mkp-scsi/for-next]
[cannot apply to jejb-scsi/for-next linus/master v6.6-rc7 next-20231027] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Karan-Tilak-Kumar/scsi-fnic-Modify-definitions-to-sync-with-VIC-firmware/20231028-060626 base: https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git for-next patch link: https://lore.kernel.org/r/20231027180302.418676-13-kartilak%40cisco.com patch subject: [PATCH v2 12/13] scsi: fnic: Add support for multiqueue (MQ) in fnic driver config: x86_64-randconfig-001-20231029 (https://download.01.org/0day-ci/archive/20231030/[email protected]/config) compiler: gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231030/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/scsi/fnic/fnic_scsi.c: In function 'fnic_queuecommand':
>> drivers/scsi/fnic/fnic_scsi.c:606:6: warning: variable 'tag' set but not used [-Wunused-but-set-variable]
int tag = 0;
^~~vim +/tag +606 drivers/scsi/fnic/fnic_scsi.c
601
602 int fnic_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *sc)
603 {
604 struct request *const rq = scsi_cmd_to_rq(sc);
605 uint32_t mqtag = 0;
> 606 int tag = 0;
607 uint16_t hwq = 0;
608
609 mqtag = blk_mq_unique_tag(rq);
610 hwq = blk_mq_unique_tag_to_hwq(mqtag);
611 tag = blk_mq_unique_tag_to_tag(mqtag);
612
613 return fnic_queuecommand_int(sc, mqtag, hwq);
614 }
615
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v2 12/13] scsi: fnic: Add support for multiqueue (MQ) in fnic driver
In reply to Karan Tilak KumarQuoted 332 lines · Karan Tilak Kumar
Implement support for MQ in fnic driver:
The block multiqueue layer issues IO to the fnic driver with an MQ tag. Use the mqtag and derive a tag from it. Derive the hardware queue from the mqtag and use it in all paths. Modify queuecommand to handle mqtag.
Replace wq and cq indices to support MQ. Replace the zeroth queue with a hardware queue. Implement spin locks on a per hardware queue basis. Replace
io_lockwith per hardware queue spinlock. Implement out of range tag checks.Allocate an
io_req_tableto track status of theio_req.Test the driver by building it, loading it,
and configuring 64 queues in UCSM. Issue IOs using Medusa on multiple fnics. Enable/disable links to exercise the abort and clean up path.Changes between v1 and v2:
Incorporate the following review comments from Bart:
Remove outdated comment,
Remove unnecessary out of range tag checks, Remove unnecessary local variable,
Modify function name.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Tested-by Karan Tilak Kumar <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 2 - drivers/scsi/fnic/fnic_io.h | 2 + drivers/scsi/fnic/fnic_main.c | 3 - drivers/scsi/fnic/fnic_scsi.c | 571 ++++++++++++++++++++-------------- 4 files changed, 346 insertions(+), 232 deletions(-)drivers/scsi/fnic/fnic.h +0
-2@@ -36,7 +36,6 @@#define FNIC_MIN_IO_REQ 256/* Min IO throttle count */ #define FNIC_MAX_IO_REQ 1024/* scsi_cmnd tag map entries */ #define FNIC_DFLT_IO_REQ 256/* Default scsi_cmnd tag map entries */ -#define FNIC_IO_LOCKS 64/* IO locks: power of 2 */ #define FNIC_DFLT_QUEUE_DEPTH 256#define FNIC_STATS_RATE_LIMIT 4/* limit rate at which stats are pulled up */ @@ -292,7 +291,6 @@ struct fnic {struct fnic_host_tag *tags;mempool_t *io_req_pool;mempool_t *io_sgl_pool[FNIC_SGL_NUM_CACHES ]; -spinlock_t io_req_lock[FNIC_IO_LOCKS ];/* locks for scsi cmnds */ unsigned int cpy_wq_base;struct work_struct link_work;drivers/scsi/fnic/fnic_io.h +2
-0@@ -52,6 +52,8 @@ struct fnic_io_req {unsigned long start_time;/* in jiffies */ struct completion *abts_done;/* completion for abts */ struct completion *dr_done;/* completion for device reset */ +unsigned int tag; +struct scsi_cmnd *sc;/* midlayer's cmd pointer */ };enum fnic_port_speeds {drivers/scsi/fnic/fnic_main.c +0
-3@@ -816,9 +816,6 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) fnic->fw_ack_index[i] =-1 ; } -for (i =0 ; i <FNIC_IO_LOCKS ; i++) -spin_lock_init (&fnic->io_req_lock[i]); - err = -ENOMEM ; fnic->io_req_pool =mempool_create_slab_pool (2 , fnic_io_req_cache);if (!fnic->io_req_pool)drivers/scsi/fnic/fnic_scsi.c +43
-53@@ -92,20 +92,6 @@ static const char *fnic_fcpio_status_to_str(unsigned int status)static void fnic_cleanup_io (struct fnic *fnic); -static inline spinlock_t *fnic_io_lock_hash (struct fnic *fnic, -struct scsi_cmnd *sc) -{ -u32 hash =scsi_cmd_to_rq (sc)->tag & (FNIC_IO_LOCKS -1 ); - -return &fnic->io_req_lock[hash]; -} - -static inline spinlock_t *fnic_io_lock_tag (struct fnic *fnic, -int tag) -{ -return &fnic->io_req_lock[tag & (FNIC_IO_LOCKS -1 )]; -} -/* * Unmap the data buffer and sense buffer for an io_req, * also unmap and free the device-private scatter/gather list. @@ -129,23 +115,23 @@ static void fnic_release_ioreq_buf(struct fnic *fnic, }/* Free up Copy Wq descriptors. Called with copy_wq lock held */ -static int free_wq_copy_descs (struct fnic *fnic,struct vnic_wq_copy *wq) +static int free_wq_copy_descs (struct fnic *fnic,struct vnic_wq_copy *wq,unsigned int hwq) {/* if no Ack received from firmware, then nothing to clean */ -if (!fnic->fw_ack_recd[0 ]) +if (!fnic->fw_ack_recd[hwq])return 1 ;/* * Update desc_available count based on number of freed descriptors * Account for wraparound */ -if (wq->to_clean_index <= fnic->fw_ack_index[0 ]) - wq->ring.desc_avail += (fnic->fw_ack_index[0 ] +if (wq->to_clean_index <= fnic->fw_ack_index[hwq]) + wq->ring.desc_avail += (fnic->fw_ack_index[hwq] - wq->to_clean_index +1 );else wq->ring.desc_avail += (wq->ring.desc_count - wq->to_clean_index - + fnic->fw_ack_index[0 ] +1 ); + + fnic->fw_ack_index[hwq] +1 );/* * just bump clean index to ack_index+1 accounting for wraparound @@ -153,10 +139,10 @@ static int free_wq_copy_descs(struct fnic *fnic, struct vnic_wq_copy *wq)* to_clean_index and fw_ack_index, both inclusive */ wq->to_clean_index = - (fnic->fw_ack_index[0 ] +1 ) % wq->ring.desc_count; + (fnic->fw_ack_index[hwq] +1 ) % wq->ring.desc_count;/* we have processed the acks received so far */ - fnic->fw_ack_recd[0 ] =0 ; + fnic->fw_ack_recd[hwq] =0 ;return 0 ; }@@ -207,7 +193,7 @@ int fnic_fw_reset_handler(struct fnic *fnic)spin_lock_irqsave (&fnic->wq_copy_lock[0 ], flags);if (vnic_wq_copy_desc_avail (wq) <= fnic->wq_copy_desc_low[0 ]) -free_wq_copy_descs (fnic, wq); +free_wq_copy_descs (fnic, wq,0 );if (!vnic_wq_copy_desc_avail (wq)) ret = -EAGAIN ;@@ -253,7 +239,7 @@ int fnic_flogi_reg_handler(struct fnic *fnic, u32 fc_id)spin_lock_irqsave (&fnic->wq_copy_lock[0 ], flags);if (vnic_wq_copy_desc_avail (wq) <= fnic->wq_copy_desc_low[0 ]) -free_wq_copy_descs (fnic, wq); +free_wq_copy_descs (fnic, wq,0 );if (!vnic_wq_copy_desc_avail (wq)) { ret = -EAGAIN ;@@ -303,7 +289,9 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic,struct vnic_wq_copy *wq, struct fnic_io_req *io_req, struct scsi_cmnd *sc, - int sg_count) + int sg_count, + uint32_t mqtag, + uint16_t hwq) { struct scatterlist *sg;struct fc_rport *rport =starget_to_rport (scsi_target (sc->device));@@ -311,7 +299,6 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic,struct host_sg_desc *desc;struct misc_stats *misc_stats = &fnic->fnic_stats.misc_stats;unsigned int i; -unsigned long intr_flags;int flags;u8 exch_flags;struct scsi_lun fc_lun;@@ -351,13 +338,10 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic,int_to_scsilun (sc->device->lun, &fc_lun);/* Enqueue the descriptor in the Copy WQ */ -spin_lock_irqsave (&fnic->wq_copy_lock[0 ], intr_flags); - -if (vnic_wq_copy_desc_avail (wq) <= fnic->wq_copy_desc_low[0 ]) -free_wq_copy_descs (fnic, wq); +if (vnic_wq_copy_desc_avail (wq) <= fnic->wq_copy_desc_low[hwq]) +free_wq_copy_descs (fnic, wq, hwq);if (unlikely (!vnic_wq_copy_desc_avail (wq))) { -spin_unlock_irqrestore (&fnic->wq_copy_lock[0 ], intr_flags);FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host,"fnic_queue_wq_copy_desc failure - no descriptors\n" );atomic64_inc (&misc_stats->io_cpwq_alloc_failures);@@ -375,7 +359,7 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic, (rp->flags & FC_RP_FLAGS_RETRY)) exch_flags |=FCPIO_ICMND_SRFLAG_RETRY ; - fnic_queue_wq_copy_desc_icmnd_16(wq, scsi_cmd_to_rq(sc)->tag, + fnic_queue_wq_copy_desc_icmnd_16(wq, mqtag, 0, exch_flags, io_req->sgl_cnt, SCSI_SENSE_BUFFERSIZE, io_req->sgl_list_pa,@@ -396,31 +380,23 @@ static inline int fnic_queue_wq_copy_desc(struct fnic *fnic,atomic64_set (&fnic->fnic_stats.fw_stats.max_fw_reqs,atomic64_read (&fnic->fnic_stats.fw_stats.active_fw_reqs)); -spin_unlock_irqrestore (&fnic->wq_copy_lock[0 ], intr_flags);return 0 ; } -/* -* fnic_queuecommand -* Routine to send a scsi cdb -* Called with host_lock held and interrupts disabled. -*/ -static int fnic_queuecommand_lck (struct scsi_cmnd *sc) +static int fnic_queuecommand_int (struct scsi_cmnd *sc,uint32_t mqtag,uint16_t hwq) {void (*done)(struct scsi_cmnd *) = scsi_done; -const int tag =scsi_cmd_to_rq (sc)->tag;struct fc_lport *lp =shost_priv (sc->device->host);struct fc_rport *rport;struct fnic_io_req *io_req =NULL ;struct fnic *fnic =lport_priv (lp);struct fnic_stats *fnic_stats = &fnic->fnic_stats;struct vnic_wq_copy *wq; -int ret; +int ret =1 ;u64 cmd_trace;int sg_count =0 ;unsigned long flags =0 ;unsigned long ptr; -spinlock_t *io_lock =NULL ;int io_lock_acquired =0 ;struct fc_rport_libfc_priv *rp;@@ -514,7 +490,7 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) sg_count =scsi_dma_map (sc);if (sg_count <0 ) {FNIC_TRACE (fnic_queuecommand, sc->device->host->host_no, - tag, sc,0 , sc->cmnd[0 ], sg_count,fnic_priv (sc)->state); + mqtag, sc,0 , sc->cmnd[0 ], sg_count,fnic_priv (sc)->state);mempool_free (io_req, fnic->io_req_pool); goto out; }@@ -549,11 +525,9 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) }/* -* Will acquire lock defore setting to IO initialized. +* Will acquire lock before setting to IO initialized. */ - - io_lock =fnic_io_lock_hash (fnic, sc); -spin_lock_irqsave (io_lock, flags); +spin_lock_irqsave (&fnic->wq_copy_lock[hwq], flags);/* initialize rest of io_req */ io_lock_acquired =1 ;@@ -562,21 +536,34 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc)fnic_priv (sc)->state =FNIC_IOREQ_CMD_PENDING ;fnic_priv (sc)->io_req = io_req;fnic_priv (sc)->flags |=FNIC_IO_INITIALIZED ; + io_req->sc = sc; + +if (fnic->sw_copy_wq[hwq].io_req_table[blk_mq_unique_tag_to_tag (mqtag)] !=NULL ) { +WARN (1 ,"fnic<%d>: %s: hwq: %d tag 0x%x already exists\n" , + fnic->fnic_num, __func__, hwq,blk_mq_unique_tag_to_tag (mqtag)); +return SCSI_MLQUEUE_HOST_BUSY ; + } + + fnic->sw_copy_wq[hwq].io_req_table[blk_mq_unique_tag_to_tag (mqtag)] = io_req; + io_req->tag = mqtag;/* create copy wq desc and enqueue it */ - wq = &fnic->hw_copy_wq[0 ]; - ret =fnic_queue_wq_copy_desc (fnic, wq, io_req, sc, sg_count); + wq = &fnic->hw_copy_wq[hwq]; +atomic64_inc (&fnic_stats->io_stats.ios[hwq]); + ret =fnic_queue_wq_copy_desc (fnic, wq, io_req, sc, sg_count, mqtag, hwq);if (ret) {/* * In case another thread cancelled the request, * refetch the pointer under the lock. */ FNIC_TRACE (fnic_queuecommand, sc->device->host->host_no, - tag, sc,0 ,0 ,0 ,fnic_flags_and_state (sc)); + mqtag, sc,0 ,0 ,0 ,fnic_flags_and_state (sc)); io_req =fnic_priv (sc)->io_req;fnic_priv (sc)->io_req =NULL ; +if (io_req) + fnic->sw_copy_wq[hwq].io_req_table[blk_mq_unique_tag_to_tag (mqtag)] =NULL ;fnic_priv (sc)->state =FNIC_IOREQ_CMD_COMPLETE ; -spin_unlock_irqrestore (io_lock, flags); +spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);if (io_req) {fnic_release_ioreq_buf (fnic, io_req, sc);mempool_free (io_req, fnic->io_req_pool);@@ -601,18 +588,30 @@ static int fnic_queuecommand_lck(struct scsi_cmnd *sc) sc->cmnd[5]);FNIC_TRACE (fnic_queuecommand, sc->device->host->host_no, - tag, sc, io_req, sg_count, cmd_trace, + mqtag, sc, io_req, sg_count, cmd_trace,fnic_flags_and_state (sc));/* if only we issued IO, will we have the io lock */ if (io_lock_acquired) -spin_unlock_irqrestore (io_lock, flags); +spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);atomic_dec (&fnic->in_flight);return ret; } -DEF_SCSI_QCMD (fnic_queuecommand) +int fnic_queuecommand (struct Scsi_Host *shost,struct scsi_cmnd *sc) +{ +struct request *const rq =scsi_cmd_to_rq (sc); +uint32_t mqtag =0 ;
drivers/scsi/fnic/fnic_scsi.c static int fnic_queuecommand_lck(struct scsi_cmnd *sc) · line 615
Quoted 10 lines
drivers/scsi/fnic/fnic_scsi.c +9
-0+int tag =0 ; +uint16_t hwq =0 ; + + mqtag =blk_mq_unique_tag (rq); + hwq =blk_mq_unique_tag_to_hwq (mqtag); + tag =blk_mq_unique_tag_to_tag (mqtag); + +return fnic_queuecommand_int (sc, mqtag, hwq); +}
That is odd.
If you pass in 'mqtag' you can derive the hardware queue from it, so you don't need to pass it in separately.
Alternative it would make sense to pass in the 'tag' value instead of 'mqtag', as this would simplify the code (seeint that you don't have to call blk_mq_unique_tag_to_tag() all over the place).
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 12/13] scsi: fnic: Add support for multiqueue (MQ) in fnic driver
Quoted 11 lines · Hannes Reinecke
Quoted 1 line · Karan Tilak Kumar
That is odd.
If you pass in 'mqtag' you can derive the hardware queue from it, so you don't need to pass it in separately.
Alternative it would make sense to pass in the 'tag' value instead of 'mqtag', as this would simplify the code (seeint that you don't have to callblk_mq_unique_tag_to_tag()all over the place).Cheers,
Hannes
The former approach sounds better.
I'll adopt it in v3.
Regards,
Karan
Karan Tilak Kumar
maintainerPATCH v2 13/13 scsi: fnic: Improve logs and add support for multiqueue (MQ)
In reply to Karan Tilak KumarImprove existing logs by adding fnic number, hardware queue, tag, and mqtag in the prints.
Add logs with the above elements for effective debugging.
--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_scsi.c | 127 ++++++++++++++++++++-------------- 2 files changed, 77 insertions(+), 52 deletions(-)
drivers/scsi/fnic/fnic.h +1 -1
@@ -27,7 +27,7 @@#define DRV_NAME "fnic"#define DRV_DESCRIPTION "Cisco FCoE HBA Driver" -#define DRV_VERSION "1.6.0.56" +#define DRV_VERSION "1.6.0.58"#define PFX DRV_NAME ": "#define DFX DRV_NAME "%d: "
drivers/scsi/fnic/fnic_scsi.c +76 -51
260 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
Quoted 18 lines · Karan Tilak Kumar
Improve existing logs by adding fnic number, hardware queue, tag, and mqtag in the prints.
Add logs with the above elements for effective debugging.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Tested-by Karan Tilak Kumar <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_scsi.c | 127 ++++++++++++++++++++-------------- 2 files changed, 77 insertions(+), 52 deletions(-)drivers/scsi/fnic/fnic.h +0
-0
drivers/scsi/fnic/fnic.h line 33
Quoted 8 lines
drivers/scsi/fnic/fnic.h +1
-1@@ -27,7 +27,7 @@#define DRV_NAME "fnic"#define DRV_DESCRIPTION "Cisco FCoE HBA Driver" -#define DRV_VERSION "1.6.0.56" +#define DRV_VERSION "1.6.0.58"#define PFX DRV_NAME ": "#define DFX DRV_NAME "%d: "
Please move this change to the last patch.
And I would wager that this change is more intrusive than a simple patch, so an update to 1.7 or even 2.0 is warranted.
Otherwise:
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
Quoted 3 lines · Hannes Reinecke
Please move this change to the last patch.
And I would wager that this change is more intrusive than a simple patch, so an update to 1.7 or even 2.0 is warranted.
The version change is in the last patch of the patchset.
Do you mean that I need to make a separate patch to increment the version number? In v3, I'll update it to 1.7.0.0.
Regards,
Karan
Re: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
Quoted 9 lines · Karan Tilak Kumar (kartilak)
Quoted 3 lines · Hannes Reinecke
Please move this change to the last patch.
And I would wager that this change is more intrusive than a simple patch, so an update to 1.7 or even 2.0 is warranted.The version change is in the last patch of the patchset.
Do you mean that I need to make a separate patch to increment the version number? In v3, I'll update it to 1.7.0.0.
Yes, please. And make this the last patch of the patch series.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
Quoted 7 lines · Hannes Reinecke
Quoted 5 lines · Karan Tilak Kumar (kartilak)
Quoted 3 lines · Hannes Reinecke
Please move this change to the last patch.
And I would wager that this change is more intrusive than a simple patch, so an update to 1.7 or even 2.0 is warranted.
Quoted 5 lines
Quoted 4 lines
The version change is in the last patch of the patchset.
Do you mean that I need to make a separate patch to increment the version number? In v3, I'll update it to 1.7.0.0.Yes, please. And make this the last patch of the patch series.
Sure, will do. I'll make this change in v3.
Regards,
Karan
Re: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
In reply to Karan Tilak KumarQuoted 288 lines · Karan Tilak Kumar
Improve existing logs by adding fnic number, hardware queue, tag, and mqtag in the prints.
Add logs with the above elements for effective debugging.Reviewed-by Sesidhar Baddela <[email protected]>Reviewed-by Arulprabhu Ponnusamy <[email protected]>Tested-by Karan Tilak Kumar <[email protected]>Signed-off-by Karan Tilak Kumar <[email protected]>--- drivers/scsi/fnic/fnic.h | 2 +- drivers/scsi/fnic/fnic_scsi.c | 127 ++++++++++++++++++++-------------- 2 files changed, 77 insertions(+), 52 deletions(-)drivers/scsi/fnic/fnic.h +1
-1@@ -27,7 +27,7 @@#define DRV_NAME "fnic"#define DRV_DESCRIPTION "Cisco FCoE HBA Driver" -#define DRV_VERSION "1.6.0.56" +#define DRV_VERSION "1.6.0.58"#define PFX DRV_NAME ": "#define DFX DRV_NAME "%d: "drivers/scsi/fnic/fnic_scsi.c +70
-45@@ -211,12 +211,14 @@ int fnic_fw_reset_handler(struct fnic *fnic)if (!ret) {atomic64_inc (&fnic->fnic_stats.reset_stats.fw_resets); -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Issued fw reset\n" ); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: Issued fw reset\n" , + fnic->fnic_num, __func__, __LINE__); }else {fnic_clear_state_flags (fnic,FNIC_FLAGS_FWRESET ); -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Failed to issue fw reset\n" ); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: Failed to issue fw reset\n" , + fnic->fnic_num, __func__, __LINE__); }return ret;@@ -265,9 +267,9 @@ int fnic_flogi_reg_handler(struct fnic *fnic, u32 fc_id) }else {fnic_queue_wq_copy_desc_flogi_reg (wq,SCSI_NO_TAG , format, fc_id, gw_mac); -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"FLOGI reg issued fcid %x map %d dest %pM\n" , - fc_id, fnic->ctlr.map_dest, gw_mac); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: FLOGI reg issued fcid 0x%x map %d dest 0x%p\n" , + fnic->fnic_num, __func__, __LINE__, fc_id, fnic->ctlr.map_dest, gw_mac); }atomic64_inc (&fnic->fnic_stats.fw_stats.active_fw_reqs);@@ -644,15 +646,16 @@ static int fnic_fcpio_fw_reset_cmpl_handler(struct fnic *fnic,if (fnic->state ==FNIC_IN_FC_TRANS_ETH_MODE ) {/* Check status of reset completion */ if (!hdr_status) { -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"reset cmpl success\n" ); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: reset cmpl success\n" , + fnic->fnic_num, __func__, __LINE__);/* Ready to send flogi out */ fnic->state =FNIC_IN_ETH_MODE ; }else { -FNIC_SCSI_DBG (KERN_DEBUG , - fnic->lport->host, -"fnic fw_reset : failed %s\n" , -fnic_fcpio_status_to_str (hdr_status)); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: reset failed with header status: %s\n" , + fnic->fnic_num, __func__, __LINE__, +fnic_fcpio_status_to_str (hdr_status));/* * Unable to change to eth mode, cannot send out flogi @@ -665,10 +668,9 @@ static int fnic_fcpio_fw_reset_cmpl_handler(struct fnic *fnic, ret =-1 ; } }else { -FNIC_SCSI_DBG (KERN_DEBUG , - fnic->lport->host, -"Unexpected state %s while processing" -" reset cmpl\n" ,fnic_state_to_str (fnic->state)); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: Unexpected state while processing reset completion: %s\n" , + fnic->fnic_num, __func__, __LINE__,fnic_state_to_str (fnic->state));atomic64_inc (&reset_stats->fw_reset_failures); ret =-1 ; }@@ -1177,9 +1179,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_indeif ((id &FNIC_TAG_ABORT ) && (id &FNIC_TAG_DEV_RST )) {/* Abort and terminate completion of device reset req */ /* REVISIT : Add asserts about various flags */ -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"dev reset abts cmpl recd. id %x status %s\n" , - id,fnic_fcpio_status_to_str (hdr_status)); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Abt/term completion received\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag, +fnic_fcpio_status_to_str (hdr_status));fnic_priv (sc)->state =FNIC_IOREQ_ABTS_COMPLETE ;fnic_priv (sc)->abts_status = hdr_status;fnic_priv (sc)->flags |=FNIC_DEV_RST_DONE ;@@ -1188,6 +1191,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_indespin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); }else if (id &FNIC_TAG_ABORT ) {/* Completion of abort cmd */ +shost_printk (KERN_DEBUG , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x Abort header status: %s\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag, +fnic_fcpio_status_to_str (hdr_status));switch (hdr_status) {case FCPIO_SUCCESS :break ;@@ -1247,9 +1254,14 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_indeif (io_req->abts_done) {complete (io_req->abts_done);spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); +shost_printk (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x Waking up abort thread\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, tag); }else {FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"abts cmpl, completing IO\n" ); +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Completing IO\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, + tag,fnic_fcpio_status_to_str (hdr_status));fnic_priv (sc)->io_req =NULL ; sc->result = (DID_ERROR <<16 ); fnic->sw_copy_wq[hwq].io_req_table[tag] =NULL ;@@ -1277,6 +1289,10 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde } }else if (id &FNIC_TAG_DEV_RST ) {/* Completion of device reset */ +shost_printk (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x DR hst: %s\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, + tag,fnic_fcpio_status_to_str (hdr_status));fnic_priv (sc)->lr_status = hdr_status;if (fnic_priv (sc)->state ==FNIC_IOREQ_ABTS_PENDING ) {spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);@@ -1286,10 +1302,9 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_indejiffies_to_msecs (jiffies - start_time), desc, 0, fnic_flags_and_state(sc));FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Terminate pending " -"dev reset cmpl recd. id %d status %s\n" , - (int )(id &FNIC_TAG_MASK ), -fnic_fcpio_status_to_str (hdr_status)); +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x hst: %s Terminate pending\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, + tag,fnic_fcpio_status_to_str (hdr_status));return ; }if (fnic_priv (sc)->flags &FNIC_DEV_RST_TIMED_OUT ) {@@ -1308,18 +1323,18 @@ static void fnic_fcpio_itmf_cmpl_handler(struct fnic *fnic, unsigned int cq_inde }fnic_priv (sc)->state =FNIC_IOREQ_CMD_COMPLETE ;fnic_priv (sc)->flags |=FNIC_DEV_RST_DONE ; -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"dev reset cmpl recd. id %d status %s\n" , - (int )(id &FNIC_TAG_MASK ), -fnic_fcpio_status_to_str (hdr_status)); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d mqtag: 0x%x tag: 0x%x hst: %s DR completion received\n" , + fnic->fnic_num, __func__, __LINE__, hwq, mqtag, + tag,fnic_fcpio_status_to_str (hdr_status));if (io_req->dr_done)complete (io_req->dr_done);spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); }else {shost_printk (KERN_ERR , fnic->lport->host, -"Unexpected itmf io state %s tag %x\n" , -fnic_ioreq_state_to_str (fnic_priv (sc)->state), id); +"%s: Unexpected itmf io state: hwq: %d tag 0x%x %s\n" , + __func__, hwq, id,fnic_ioreq_state_to_str (fnic_priv (sc)->state));spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); }@@ -1470,9 +1485,9 @@ static bool fnic_cleanup_io_iter(struct scsi_cmnd *sc, void *data)mempool_free (io_req, fnic->io_req_pool); sc->result =DID_TRANSPORT_DISRUPTED <<16 ; -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"fnic_cleanup_io: tag:0x%x : sc:0x%p duration = %lu DID_TRANSPORT_DISRUPTED\n" , - tag, sc, jiffies - start_time); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: mqtag:0x%x tag: 0x%x sc:0x%p duration = %lu DID_TRANSPORT_DISRUPTED\n" , + fnic->fnic_num, __func__, __LINE__, mqtag, tag, sc, (jiffies - start_time));if (atomic64_read (&fnic->io_cmpl_skip))atomic64_dec (&fnic->io_cmpl_skip);@@ -1641,9 +1656,9 @@ static bool fnic_rport_abort_io_iter(struct scsi_cmnd *sc, void *data)if ((fnic_priv (sc)->flags &FNIC_DEVICE_RESET ) && !(fnic_priv (sc)->flags &FNIC_DEV_RST_ISSUED )) { -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"fnic_rport_exch_reset dev rst not pending sc 0x%p\n" , - sc); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d abt_tag: 0x%x flags: 0x%x Device reset is not pending\n" , + fnic->fnic_num, __func__, __LINE__, hwq, abt_tag,fnic_priv (sc)->flags);spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);return true; }@@ -1699,6 +1714,9 @@ static bool fnic_rport_abort_io_iter(struct scsi_cmnd *sc, void *data)* lun reset */ spin_lock_irqsave (&fnic->wq_copy_lock[hwq], flags); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d abt_tag: 0x%x flags: 0x%x Queuing abort failed\n" , + fnic->fnic_num, __func__, __LINE__, hwq, abt_tag,fnic_priv (sc)->flags);if (fnic_priv (sc)->state ==FNIC_IOREQ_ABTS_PENDING )fnic_priv (sc)->state = old_ioreq_state;spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);@@ -1869,8 +1887,9 @@ int fnic_abort_cmd(struct scsi_cmnd *sc)else atomic64_inc (&abts_stats->abort_issued_greater_than_60_sec); -FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, -"CBD Opcode: %02x Abort issued time: %lu msec\n" , sc->cmnd[0 ], abt_issued_time); +FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, +"fnic<%d>: %s: CDB Opcode: 0x%02x Abort issued time: %lu msec\n" , + fnic->fnic_num, __func__, sc->cmnd[0 ], abt_issued_time);/* * Command is still pending, need to abort it * If the firmware completes the command after this point, @@ -1959,8 +1978,9 @@ int fnic_abort_cmd(struct scsi_cmnd *sc)if (!(fnic_priv (sc)->flags & (FNIC_IO_ABORTED |FNIC_IO_DONE ))) {spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Issuing Host reset due to out of order IO\n" ); +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: Issuing host reset due to out of order IO\n" , + fnic->fnic_num, __func__); ret =FAILED ; goto fnic_abort_cmd_end;@@ -2167,6 +2187,9 @@ static bool fnic_pending_aborts_iter(struct scsi_cmnd *sc, void *data)fnic_priv (sc)->state = old_ioreq_state;spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags); iter_data->ret =FAILED ; +FNIC_SCSI_DBG (KERN_ERR , fnic->lport->host, +"fnic<%d>: %s: %d: hwq: %d abt_tag: 0x%lx Abort could not be queued\n" , + fnic->fnic_num, __func__, __LINE__, hwq, abt_tag);return false; }else {spin_lock_irqsave (&fnic->wq_copy_lock[hwq], flags);@@ -2300,8 +2323,9 @@ int fnic_device_reset(struct scsi_cmnd *sc) rport =starget_to_rport (scsi_target (sc->device));FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Device reset called FCID 0x%x, LUN 0x%llx sc 0x%p\n" , - rport->port_id, sc->device->lun, sc); +"fnic<%d>: %s: %d: fcid: 0x%x lun: 0x%llx hwq: %d mqtag: 0x%x flags: 0x%x Device reset\n" , + fnic->fnic_num, __func__, __LINE__, rport->port_id, sc->device->lun, hwq, mqtag, +fnic_priv (sc)->flags);if (lp->state !=LPORT_ST_READY || !(lp->link_up)) goto fnic_device_reset_end;@@ -2536,8 +2560,9 @@ int fnic_reset(struct Scsi_Host *shost) fnic =lport_priv (lp); reset_stats = &fnic->fnic_stats.reset_stats; -FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"fnic_reset called\n" ); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: Issuing fnic reset\n" , + fnic->fnic_num, __func__, __LINE__);atomic64_inc (&reset_stats->fnic_resets);@@ -2547,10 +2572,9 @@ int fnic_reset(struct Scsi_Host *shost)*/ ret =fc_lport_reset (lp);
drivers/scsi/fnic/fnic_scsi.c int fnic_reset(struct Scsi_Host *shost) · line 2580
Quoted 10 lines
drivers/scsi/fnic/fnic_scsi.c +3
-4-FNIC_SCSI_DBG (KERN_DEBUG , fnic->lport->host, -"Returning from fnic reset %s\n" , - (ret ==0 ) ? -"SUCCESS" :"FAILED" ); +FNIC_SCSI_DBG (KERN_INFO , fnic->lport->host, +"fnic<%d>: %s: %d: Returning from fnic reset with: %s\n" , + fnic->fnic_num, __func__, __LINE__, (ret ==0 ) ?"SUCCESS" :"FAILED" );if (ret ==0 ) atomic64_inc(&reset_stats->fnic_reset_completio to a > @@ -2766,8 +2790,9 @@ static bool fnic_abts_pending_iter(struct
scsi_cmnd *sc, void *data)
drivers/scsi/fnic/fnic_scsi.c static bool fnic_abts_pending_iter(struct scsi_cmnd *sc, void *data) · line 2798
Quoted 11 lines
* belongs to the LUN that we are resetting */
FNIC_SCSI_DBG(KERN_INFO,fnic->lport->host,drivers/scsi/fnic/fnic_scsi.c +3
-2-"Found IO in %s on lun\n" , - fnic_ioreq_state_to_str(fnic_priv(sc)->state)); +"fnic<%d>: %s: %d: hwq: %d tag: 0x%x Found IO in state: %s on lun\n" , + fnic->fnic_num, __func__, __LINE__, hwq, tag, + fnic_ioreq_state_to_str(fnic_priv(sc)->state)); cmd_state =fnic_priv (sc)->state;spin_unlock_irqrestore (&fnic->wq_copy_lock[hwq], flags);if (cmd_state ==FNIC_IOREQ_ABTS_PENDING )
See my comments to the previous patch. Please update FNIC_SCSI_DBG() to
accept an 'fnic' parameter instead of 'fnic->lport->host', and generate the 'fnic<%d>' here in the macro.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Karan Tilak Kumar (kartilak)
maintainerRE: [PATCH v2 13/13] scsi: fnic: Improve logs and add support for multiqueue (MQ)
Quoted 3 lines · Hannes Reinecke
See my comments to the previous patch. Please update
FNIC_SCSI_DBG()to accept an 'fnic' parameter instead of 'fnic->lport->host', and generate the 'fnic<%d>' here in the macro.
Sure, I'll make suitable changes in v3.
Regards,
Karan
Quoted 4 lines · Karan Tilak Kumar
The number of IO queues to be used is stored in a configuration file by the VIC firmware. The fNIC driver reads the configuration file and sets the number of queues to be used. Previously, the driver was hard-coded to use only one queue.
In looking at commit aec95e3a8ded ("scsi: fnic: Refactor code in fnic probe to initialize SCSI layer"), support to set shost->nr_hw_queues > 1 to enable MQ seems to have sneaked in already - what's going on there?
John
Karan Tilak Kumar (kartilak)
maintainerQuoted 7 lines · John Garry
Quoted 4 lines · Karan Tilak Kumar
The number of IO queues to be used is stored in a configuration file by the VIC firmware. The fNIC driver reads the configuration file and sets the number of queues to be used. Previously, the driver was hard-coded to use only one queue.
Quoted 5 lines
In looking at commit
aec95e3a8ded("scsi: fnic: Refactor code in fnic probe to initialize SCSI layer"), support to setshost->nr_hw_queues> 1 to enable MQ seems to have sneaked in already - what's going on there?John
Thanks for your review comments, John.
This was a defensive check and code to show that fnic does not have support for MQ when the user tries to configure multiple IO queues.
The support for MQ is being added only now.
Regards,
Karan