Skip to contentSkip to conversation

Re-design MSI/MSI-X interrupts enablement pattern

RFC PATCH NAK Alexander Gordeev 19 participants 150 messages Oct 2, 2013 – Oct 15, 2013

Subsystems (40)
lore
Share & tools
lore
#0 lore raw

cover Re-design MSI/MSI-X interrupts enablement pattern

This series is against "next" branch in Bjorn's repo:
git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git

Currently pci_enable_msi_block() and pci_enable_msix() interfaces return a error code in case of failure, 0 in case of success and a positive value which indicates the number of MSI-X/MSI interrupts that could have been allocated. The latter value should be passed to a repeated call to the interfaces until a failure or success:

	for (i = 0; i < FOO_DRIVER_MAXIMUM_NVEC; i++)
		adapter->msix_entries[i].entry = i;

	while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
		rc = pci_enable_msix(adapter->pdev,
				     adapter->msix_entries, nvec);
		if (rc > 0)
			nvec = rc;
		else
			return rc;
	}

	return -ENOSPC;

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

As result, device drivers will cease to use the overcomplicated repeated fallbacks technique and resort to a straightforward pattern - determine the number of MSI/MSI-X interrupts required before calling pci_enable_msi_block() and pci_enable_msix() interfaces:

	rc = pci_msix_table_size(adapter->pdev);
	if (rc < 0)
		return rc;

	nvec = min(nvec, rc);
	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
		return -ENOSPC;

	for (i = 0; i < nvec; i++)
		adapter->msix_entries[i].entry = i;

	rc = pci_enable_msix(adapter->pdev,
			     adapter->msix_entries, nvec);
	return rc;

Device drivers will use their knowledge of underlying hardware to determine the number of MSI/MSI-X interrupts required.

The simplest case would be requesting all available interrupts - to obtain that value device drivers will use pci_get_msi_cap() interface for MSI and pci_msix_table_size() for MSI-X.

More complex cases would entail matching device capabilities to the system environment, i.e. limiting number of hardware queues (and hence associated MSI/MSI-X interrupts) to the number of online CPUs.

Device drivers using MSI/MSI-X could be divided in three groups:
- drivers that request a hardcoded number of interrupts;
- drivers that request a number of interrupts using one call to pci_enable_msix() and then enable MSI/MSI-X using a follow-up to pci_enable_msix();
- drivers that fully follow the guidelines and repeatedly call pci_enable_msix() in a loop;
This series converts to the new technique second and third groups.

To simplify device drivers code review I tried to make as little changes as possible - the scope of this series is an introduction of the new technique rather than clean-up effort for all drivers affected.

The testing was very limited - I ensured successful booting on all affected architectures except MIPS and operation of few devices with and without pci=nomsi kernel parameter.

There is a ongoing discussion about impact of this update on PowerPC pSeries platform. I am going to incorporate the outcome of this discussion into the next version. Yet, the rest of the platforms and the vast majority of device drivers already can start getting initial reviews.

Patches 5,6,8 - update of the generic MSI code
Patch 7 - update of architectures affected
Patches 9-77 - bugfixes and updates of device drivers affected

The tree could be found in "pci-next-msi-v1" branch in repo:
https://github.com/a-gordeev/linux.git

Alexander Gordeev (77):
PCI/MSI: Fix return value when populate_msi_sysfs() failed PCI/MSI/PPC: Fix wrong RTAS error code reporting PCI/MSI/s390: Fix single MSI only check
PCI/MSI/s390: Remove superfluous check of MSI type PCI/MSI: Convert pci_msix_table_size() to a public interface PCI/MSI: Factor out pci_get_msi_cap() interface
PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern PCI/MSI: Get rid of pci_enable_msi_block_auto() interface
ahci: Update MSI/MSI-X interrupts enablement code
ahci: Check MRSM bit when multiple MSIs enabled
benet: Return -ENOSPC when not enough MSI-Xs available
benet: Update MSI/MSI-X interrupts enablement code
bna: Update MSI/MSI-X interrupts enablement code
bnx2x: Update MSI/MSI-X interrupts enablement code
bnx2: Update MSI/MSI-X interrupts enablement code
cciss: Update MSI/MSI-X interrupts enablement code
cciss: Update a misleading comment on interrupt usage
cciss: Fallback to single MSI mode in case MSI-X failed
csiostor: Do not call pci_disable_msix() if pci_enable_msix() failed
csiostor: Return -ENOSPC when not enough MSI-X vectors available
csiostor: Update MSI/MSI-X interrupts enablement code
cxgb3: Do not call pci_disable_msix() if pci_enable_msix() failed
cxgb3: Return -ENOSPC when not enough MSI-X vectors available
cxgb3: Update MSI/MSI-X interrupts enablement code
cxgb4: Return -ENOSPC when not enough MSI-X vectors available
cxgb4: Update MSI/MSI-X interrupts enablement code
cxgb4vf: Do not call pci_disable_msix() if pci_enable_msix() failed
cxgb4vf: Return -ENOSPC when not enough MSI-X vectors available
cxgb4vf: Update MSI/MSI-X interrupts enablement code
hpsa: Update a misleading comment on interrupt usage
hpsa: Update MSI/MSI-X interrupts enablement code
hpsa: Fallback to single MSI mode in case MSI-X failed
ioat: Disable MSI-X in case request of IRQ failed
ioat: Update MSI/MSI-X interrupts enablement code
ipr: Do not call pci_disable_msi/msix() if pci_enable_msi/msix()
failed
ipr: Enable MSI-X when IPR_USE_MSIX type is set, not IPR_USE_MSI
ipr: Update MSI/MSI-X interrupts enablement code
ixgbe: Update MSI/MSI-X interrupts enablement code
ixgbevf: Return -ENOSPC when not enough MSI-X vectors available
ixgbevf: Update MSI/MSI-X interrupts enablement code
lpfc: Do not call pci_disable_msix() if pci_enable_msix() failed
lpfc: Update MSI/MSI-X interrupts enablement code
lpfc: Return -ENOSPC when not enough MSI-X vectors available
lpfc: Make MSI-X initialization routine more readable
megaraid: Update MSI/MSI-X interrupts enablement code
mlx4: Update MSI/MSI-X interrupts enablement code
mlx5: Fix memory leak in case not enough MSI-X vectors available
mlx5: Return -ENOSPC when not enough MSI-X vectors available
mlx5: Fix minimum number of MSI-Xs
mlx5: Update MSI/MSI-X interrupts enablement code
mthca: Update MSI/MSI-X interrupts enablement code
niu: Update MSI/MSI-X interrupts enablement code
ntb: Fix missed call to pci_enable_msix()
ntb: Ensure number of MSIs on SNB is enough for the link interrupt
ntb: Update MSI/MSI-X interrupts enablement code
nvme: Update MSI/MSI-X interrupts enablement code
pmcraid: Update MSI/MSI-X interrupts enablement code
qib: Update MSI/MSI-X interrupts enablement code
qla2xxx: Update MSI/MSI-X interrupts enablement code
qlcnic: Return -ENOSPC when not enough MSI-X vectors available
qlogic: Return -EINVAL in case MSI-X is not supported
qlcnic: Remove redundant return operator
qlcnic: Update MSI/MSI-X interrupts enablement code
qlcnic: Make MSI-X initialization routine bit more readable
qlge: Remove a redundant assignment
qlge: Update MSI/MSI-X interrupts enablement code
rapidio: Update MSI/MSI-X interrupts enablement code
sfc: Update MSI/MSI-X interrupts enablement code
tg3: Update MSI/MSI-X interrupts enablement code
vmci: Update MSI/MSI-X interrupts enablement code
vmxnet3: Return -EINVAL if number of requested MSI-Xs is not enough
vmxnet3: Fixup a weird loop exit
vmxnet3: Return -ENOSPC when not enough MSI-X vectors available
vmxnet3: Limit number of rx queues to 1 if per-queue MSI-Xs failed
vmxnet3: Update MSI/MSI-X interrupts enablement code
vxge: Sanitize MSI-X allocation routine error codes
vxge: Update MSI/MSI-X interrupts enablement code

 Documentation/PCI/MSI-HOWTO.txt                    |  123 +++++++++++---------
 arch/mips/pci/msi-octeon.c                         |    2 +-
 arch/powerpc/kernel/msi.c                          |    2 +-
 arch/powerpc/platforms/pseries/msi.c               |    4 +-
 arch/s390/pci/pci.c                                |    2 +-
 arch/x86/kernel/apic/io_apic.c                     |    2 +-
 drivers/ata/ahci.c                                 |   71 ++++++++---
 drivers/ata/ahci.h                                 |    1 +
 drivers/block/cciss.c                              |   22 ++--
 drivers/block/nvme-core.c                          |   48 ++++----
 drivers/dma/ioat/dma.c                             |   11 ++-
 drivers/infiniband/hw/mthca/mthca_main.c           |   16 ++-
 drivers/infiniband/hw/qib/qib_pcie.c               |    4 -
 drivers/misc/vmw_vmci/vmci_guest.c                 |   22 +++-
 drivers/net/ethernet/broadcom/bnx2.c               |   27 +++--
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c    |   54 ++++-----
 drivers/net/ethernet/broadcom/tg3.c                |   24 ++--
 drivers/net/ethernet/brocade/bna/bnad.c            |   34 +++---
 drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c    |   32 +++---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c    |   62 ++++++----
 .../net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c    |   49 +++++---
 drivers/net/ethernet/emulex/benet/be_main.c        |   36 +++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c       |   62 +++++-----
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c  |   18 +--
 drivers/net/ethernet/mellanox/mlx4/main.c          |   17 ++--
 drivers/net/ethernet/mellanox/mlx5/core/main.c     |   17 ++--
 drivers/net/ethernet/neterion/vxge/vxge-main.c     |   38 +++----
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c   |  108 +++++++++--------
 drivers/net/ethernet/qlogic/qlge/qlge_main.c       |   40 +++----
 drivers/net/ethernet/sfc/efx.c                     |   18 ++-
 drivers/net/ethernet/sun/niu.c                     |   20 ++--
 drivers/net/vmxnet3/vmxnet3_drv.c                  |   84 +++++++-------
 drivers/ntb/ntb_hw.c                               |   37 ++----
 drivers/ntb/ntb_hw.h                               |    2 -
 drivers/pci/msi.c                                  |   93 +++++----------
 drivers/pci/pcie/portdrv_core.c                    |    2 +
 drivers/rapidio/devices/tsi721.c                   |   27 +++--
 drivers/scsi/csiostor/csio_isr.c                   |   20 ++--
 drivers/scsi/hpsa.c                                |   35 +++---
 drivers/scsi/ipr.c                                 |   52 ++++-----
 drivers/scsi/lpfc/lpfc_init.c                      |   40 ++++---
 drivers/scsi/megaraid/megaraid_sas_base.c          |   20 ++--
 drivers/scsi/pmcraid.c                             |   23 ++--
 drivers/scsi/qla2xxx/qla_isr.c                     |   18 ++-
 include/linux/pci.h                                |    7 +-
 45 files changed, 744 insertions(+), 702 deletions(-)
Signature · 6 lines
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#1 lore raw

RFC PATCH 1/77 PCI/MSI: Fix return value when populate_msi_sysfs() failed

---
 drivers/pci/msi.c |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)
drivers/pci/msi.c +5 -6
index d5f90d6..b43f391 100644
@@ -719,7 +719,7 @@ static int msix_capability_init(struct pci_dev *dev,
 
 	ret = arch_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
 	if (ret)
-		goto error;
+		goto out_avail;
 
 	/*
 	 * Some devices require MSI-X to be enabled before we can touch the
@@ -732,10 +732,8 @@ static int msix_capability_init(struct pci_dev *dev,
 	msix_program_entries(dev, entries);
 
 	ret = populate_msi_sysfs(dev);
-	if (ret) {
-		ret = 0;
-		goto error;
-	}
+	if (ret)
+		goto out_free;
 
 	/* Set MSI-X enabled bits and unmask the function */
 	pci_intx_for_msi(dev, 0);
@@ -746,7 +744,7 @@ static int msix_capability_init(struct pci_dev *dev,
 
 	return 0;
 
-error:
+out_avail:
 	if (ret < 0) {
 		/*
 		 * If we had some success, report the number of irqs
@@ -763,6 +761,7 @@ error:
 			ret = avail;
 	}
 
+out_free:
 	free_msi_irqs(dev);
 
 	return ret;
Signature · 1 line
1.7.7.6
#5 lore raw

RFC PATCH 2/77 PCI/MSI/PPC: Fix wrong RTAS error code reporting

In reply to Alexander Gordeev
---
 arch/powerpc/platforms/pseries/msi.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/powerpc/platforms/pseries/msi.c +1 -1
index 6d2f0ab..009ec73 100644
@@ -467,7 +467,7 @@ again:
 	list_for_each_entry(entry, &pdev->msi_list, list) {
 		hwirq = rtas_query_irq_number(pdn, i++);
 		if (hwirq < 0) {
-			pr_debug("rtas_msi: error (%d) getting hwirq\n", rc);
+			pr_debug("rtas_msi: error (%d) getting hwirq\n", hwirq);
 			return hwirq;
 		}
 
Signature · 1 line
1.7.7.6
#6 lore raw

RFC PATCH 3/77 PCI/MSI/s390: Fix single MSI only check

In reply to Alexander Gordeev

Multiple MSIs have never been supported on s390 architecture, but the platform code fails to report single MSI only.

---
 arch/s390/pci/pci.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/s390/pci/pci.c +2 -0
index f17a834..c79c6e4 100644
@@ -427,6 +427,8 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 	pr_debug("%s: requesting %d MSI-X interrupts...", __func__, nvec);
 	if (type != PCI_CAP_ID_MSIX && type != PCI_CAP_ID_MSI)
 		return -EINVAL;
+	if (type == PCI_CAP_ID_MSI && nvec > 1)
+		return 1;
 	msi_vecs = min(nvec, ZPCI_MSI_VEC_MAX);
 	msi_vecs = min_t(unsigned int, msi_vecs, CONFIG_PCI_NR_MSI);
 
Signature · 1 line
1.7.7.6
#8 lore raw

RFC PATCH 4/77 PCI/MSI/s390: Remove superfluous check of MSI type

In reply to Alexander Gordeev

arch_setup_msi_irqs() hook can only be called from the generic MSI code which ensures correct MSI type parameter.

---
 arch/s390/pci/pci.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)
arch/s390/pci/pci.c +0 -2
index c79c6e4..61a3c2c 100644
@@ -425,8 +425,6 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 	int rc;
 
 	pr_debug("%s: requesting %d MSI-X interrupts...", __func__, nvec);
-	if (type != PCI_CAP_ID_MSIX && type != PCI_CAP_ID_MSI)
-		return -EINVAL;
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
 		return 1;
 	msi_vecs = min(nvec, ZPCI_MSI_VEC_MAX);
Signature · 1 line
1.7.7.6
#11 lore raw

RFC PATCH 5/77 PCI/MSI: Convert pci_msix_table_size() to a public interface

In reply to Alexander Gordeev

Make pci_msix_table_size() to return a error code if the device does not support MSI-X. This update is needed to facilitate a forthcoming re-design MSI/MSI-X interrupts enabling pattern.

Device drivers will use this interface to obtain maximum number of MSI-X interrupts the device supports and use that value in the following call to pci_enable_msix() interface.

---
 Documentation/PCI/MSI-HOWTO.txt |   13 +++++++++++++
 drivers/pci/msi.c               |    5 ++++-
 drivers/pci/pcie/portdrv_core.c |    2 ++
 3 files changed, 19 insertions(+), 1 deletions(-)
Documentation/PCI/MSI-HOWTO.txt +13 -0
index a091780..35b2d64 100644
@@ -255,6 +255,19 @@ MSI-X Table.  This address is mapped by the PCI subsystem, and should not
 be accessed directly by the device driver.  If the driver wishes to
 mask or unmask an interrupt, it should call disable_irq() / enable_irq().
 
+4.3.4 pci_msix_table_size
+
+int pci_msix_table_size(struct pci_dev *dev)
+
+This function could be used to retrieve number of entries in the device
+MSI-X table.
+
+If this function returns a negative number, it indicates the device is
+not capable of sending MSI-Xs.
+
+If this function returns a positive number, it indicates the maximum
+number of MSI-X interrupt vectors that could be allocated.
+
 4.4 Handling devices implementing both MSI and MSI-X capabilities
 
 If a device implements both MSI and MSI-X capabilities, it can
drivers/pci/msi.c +4 -1
index b43f391..875c353 100644
@@ -928,11 +928,12 @@ int pci_msix_table_size(struct pci_dev *dev)
 	u16 control;
 
 	if (!dev->msix_cap)
-		return 0;
+		return -EINVAL;
 
 	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
 	return msix_table_size(control);
 }
+EXPORT_SYMBOL(pci_msix_table_size);
 
 /**
  * pci_enable_msix - configure device's MSI-X capability structure
@@ -962,6 +963,8 @@ int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
 		return status;
 
 	nr_entries = pci_msix_table_size(dev);
+	if (nr_entries < 0)
+		return nr_entries;
 	if (nvec > nr_entries)
 		return nr_entries;
 
drivers/pci/pcie/portdrv_core.c +2 -0
index 31063ac..b4d86eb 100644
@@ -80,6 +80,8 @@ static int pcie_port_enable_msix(struct pci_dev *dev, int *vectors, int mask)
 	u32 reg32;
 
 	nr_entries = pci_msix_table_size(dev);
+	if (nr_entries < 0)
+		return nr_entries;
 	if (!nr_entries)
 		return -EINVAL;
 	if (nr_entries > PCIE_PORT_MAX_MSIX_ENTRIES)
Signature · 1 line
1.7.7.6
#14 lore raw

RFC PATCH 6/77 PCI/MSI: Factor out pci_get_msi_cap() interface

In reply to Alexander Gordeev

This update is needed to facilitate a forthcoming re-design MSI/MSI-X interrupts enabling pattern.

Device drivers will use this interface to obtain maximum number of MSI interrupts the device supports and use that value in the following call to pci_enable_msi_block() interface.

---
 Documentation/PCI/MSI-HOWTO.txt |   15 +++++++++++++++
 drivers/pci/msi.c               |   33 +++++++++++++++++++++------------
 include/linux/pci.h             |    6 ++++++
 3 files changed, 42 insertions(+), 12 deletions(-)
Documentation/PCI/MSI-HOWTO.txt +15 -0
index 35b2d64..1f37ce2 100644
@@ -169,6 +169,21 @@ on any interrupt for which it previously called request_irq().
 Failure to do so results in a BUG_ON(), leaving the device with
 MSI enabled and thus leaking its vector.
 
+4.2.5 pci_get_msi_cap
+
+int pci_get_msi_cap(struct pci_dev *dev)
+
+This function could be used to retrieve the number of MSI vectors the
+device requested (via the Multiple Message Capable register). The MSI
+specification only allows the returned value to be a power of two,
+up to a maximum of 2^5 (32).
+
+If this function returns a negative number, it indicates the device is
+not capable of sending MSIs.
+
+If this function returns a positive number, it indicates the maximum
+number of MSI interrupt vectors that could be allocated.
+
 4.3 Using MSI-X
 
 The MSI-X capability is much more flexible than the MSI capability.
drivers/pci/msi.c +21 -12
index 875c353..ca59bfc 100644
@@ -812,6 +812,21 @@ static int pci_msi_check_device(struct pci_dev *dev, int nvec, int type)
 	return 0;
 }
 
+int pci_get_msi_cap(struct pci_dev *dev)
+{
+	int ret;
+	u16 msgctl;
+
+	if (!dev->msi_cap)
+		return -EINVAL;
+
+	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &msgctl);
+	ret = 1 << ((msgctl & PCI_MSI_FLAGS_QMASK) >> 1);
+
+	return ret;
+}
+EXPORT_SYMBOL(pci_get_msi_cap);
+
 /**
  * pci_enable_msi_block - configure device's MSI capability structure
  * @dev: device to configure
@@ -828,13 +843,10 @@ static int pci_msi_check_device(struct pci_dev *dev, int nvec, int type)
 int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec)
 {
 	int status, maxvec;
-	u16 msgctl;
 
-	if (!dev->msi_cap)
-		return -EINVAL;
-
-	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &msgctl);
-	maxvec = 1 << ((msgctl & PCI_MSI_FLAGS_QMASK) >> 1);
+	maxvec = pci_get_msi_cap(dev);
+	if (maxvec < 0)
+		return maxvec;
 	if (nvec > maxvec)
 		return maxvec;
 
@@ -859,13 +871,10 @@ EXPORT_SYMBOL(pci_enable_msi_block);
 int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *maxvec)
 {
 	int ret, nvec;
-	u16 msgctl;
 
-	if (!dev->msi_cap)
-		return -EINVAL;
-
-	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &msgctl);
-	ret = 1 << ((msgctl & PCI_MSI_FLAGS_QMASK) >> 1);
+	ret = pci_get_msi_cap(dev);
+	if (ret < 0)
+		return ret;
 
 	if (maxvec)
 		*maxvec = ret;
include/linux/pci.h +6 -0
index da172f9..2fa92ef 100644
@@ -1144,6 +1144,11 @@ struct msix_entry {
 
 
 #ifndef CONFIG_PCI_MSI
+static inline int pci_get_msi_cap(struct pci_dev *dev)
+{
+	return -1;
+}
+
 static inline int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec)
 {
 	return -1;
@@ -1185,6 +1190,7 @@ static inline int pci_msi_enabled(void)
 	return 0;
 }
 #else
+int pci_get_msi_cap(struct pci_dev *dev);
 int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec);
 int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *maxvec);
 void pci_msi_shutdown(struct pci_dev *dev);
Signature · 1 line
1.7.7.6
#17 lore raw

RFC PATCH 7/77 PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern

In reply to Alexander Gordeev

Currently pci_enable_msi_block() and pci_enable_msix() interfaces return a error code in case of failure, 0 in case of success and a positive value which indicates the number of MSI-X/MSI interrupts that could have been allocated. The latter value should be passed to a repeated call to the interfaces until a failure or success.

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

As result, device drivers will cease to use the overcomplicated repeated fallbacks technique and resort to a straightforward pattern - determine the number of MSI/MSI-X interrupts required before calling pci_enable_msix() and pci_enable_msi_block() interfaces.

Device drivers will use their knowledge of underlying hardware to determine the number of MSI/MSI-X interrupts required.

The simplest case would be requesting all available interrupts - to obtain that value device drivers will use pci_get_msi_cap() interface for MSI and pci_msix_table_size() for MSI-X.

More complex cases would entail matching device capabilities with the system environment, i.e. limiting number of hardware queues (and hence associated MSI/MSI-X interrupts) to the number of online CPUs.

---
 Documentation/PCI/MSI-HOWTO.txt      |   71 ++++++++++++++++++---------------
 arch/mips/pci/msi-octeon.c           |    2 +-
 arch/powerpc/kernel/msi.c            |    2 +-
 arch/powerpc/platforms/pseries/msi.c |    2 +-
 arch/s390/pci/pci.c                  |    2 +-
 arch/x86/kernel/apic/io_apic.c       |    2 +-
 drivers/pci/msi.c                    |   52 +++++++------------------
 7 files changed, 58 insertions(+), 75 deletions(-)
Documentation/PCI/MSI-HOWTO.txt +39 -32
index 1f37ce2..40abcfb 100644
@@ -111,21 +111,27 @@ the device are in the range dev->irq to dev->irq + count - 1.
 
 If this function returns a negative number, it indicates an error and
 the driver should not attempt to request any more MSI interrupts for
-this device.  If this function returns a positive number, it is
-less than 'count' and indicates the number of interrupts that could have
-been allocated.  In neither case is the irq value updated or the device
-switched into MSI mode.
-
-The device driver must decide what action to take if
-pci_enable_msi_block() returns a value less than the number requested.
-For instance, the driver could still make use of fewer interrupts;
-in this case the driver should call pci_enable_msi_block()
-again.  Note that it is not guaranteed to succeed, even when the
-'count' has been reduced to the value returned from a previous call to
-pci_enable_msi_block().  This is because there are multiple constraints
-on the number of vectors that can be allocated; pci_enable_msi_block()
-returns as soon as it finds any constraint that doesn't allow the
-call to succeed.
+this device.
+
+Device drivers should normally call pci_get_msi_cap() function before
+calling this function to determine maximum number of MSI interrupts
+a device can send.
+
+A sequence to achieve that might look like:
+
+static int foo_driver_enable_msi(struct foo_adapter *adapter, int nvec)
+{
+	rc = pci_get_msi_cap(adapter->pdev);
+	if (rc < 0)
+		return rc;
+
+	nvec = min(nvec, rc);
+	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
+		return -ENOSPC;
+
+	rc = pci_enable_msi_block(adapter->pdev, nvec);
+	return rc;
+}
 
 4.2.3 pci_enable_msi_block_auto
 
@@ -218,9 +224,7 @@ interrupts assigned to the MSI-X vectors so it can free them again later.
 
 If this function returns a negative number, it indicates an error and
 the driver should not attempt to allocate any more MSI-X interrupts for
-this device.  If it returns a positive number, it indicates the maximum
-number of interrupt vectors that could have been allocated. See example
-below.
+this device.
 
 This function, in contrast with pci_enable_msi(), does not adjust
 dev->irq.  The device will not generate interrupts for this interrupt
@@ -229,24 +233,27 @@ number once MSI-X is enabled.
 Device drivers should normally call this function once per device
 during the initialization phase.
 
-It is ideal if drivers can cope with a variable number of MSI-X interrupts;
-there are many reasons why the platform may not be able to provide the
-exact number that a driver asks for.
+Device drivers should normally call pci_msix_table_size() function before
+calling this function to determine maximum number of MSI-X interrupts
+a device can send.
 
-A request loop to achieve that might look like:
+A sequence to achieve that might look like:
 
 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec)
 {
-	while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
-		rc = pci_enable_msix(adapter->pdev,
-				     adapter->msix_entries, nvec);
-		if (rc > 0)
-			nvec = rc;
-		else
-			return rc;
-	}
-
-	return -ENOSPC;
+	rc = pci_msix_table_size(adapter->pdev);
+	if (rc < 0)
+		return rc;
+
+	nvec = min(nvec, rc);
+	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
+		return -ENOSPC;
+
+	for (i = 0; i < nvec; i++)
+		adapter->msix_entries[i].entry = i;
+
+	rc = pci_enable_msix(adapter->pdev, adapter->msix_entries, nvec);
+	return rc;
 }
 
 4.3.2 pci_disable_msix
arch/mips/pci/msi-octeon.c +1 -1
index d37be36..0ee5f4d 100644
@@ -193,7 +193,7 @@ int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 	 * override arch_setup_msi_irqs()
 	 */
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
-		return 1;
+		return -EINVAL;
 
 	list_for_each_entry(entry, &dev->msi_list, list) {
 		ret = arch_setup_msi_irq(dev, entry);
arch/powerpc/kernel/msi.c +1 -1
index 8bbc12d..36d70b9 100644
@@ -22,7 +22,7 @@ int arch_msi_check_device(struct pci_dev* dev, int nvec, int type)
 
 	/* PowerPC doesn't support multiple MSI yet */
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
-		return 1;
+		return -EINVAL;
 
 	if (ppc_md.msi_check_device) {
 		pr_debug("msi: Using platform check routine.\n");
arch/powerpc/platforms/pseries/msi.c +1 -1
index 009ec73..89648c1 100644
@@ -348,7 +348,7 @@ static int rtas_msi_check_device(struct pci_dev *pdev, int nvec, int type)
 	quota = msi_quota_for_device(pdev, nvec);
 
 	if (quota && quota < nvec)
-		return quota;
+		return -ENOSPC;
 
 	return 0;
 }
arch/s390/pci/pci.c +1 -1
index 61a3c2c..45a1875 100644
@@ -426,7 +426,7 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 
 	pr_debug("%s: requesting %d MSI-X interrupts...", __func__, nvec);
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
-		return 1;
+		return -EINVAL;
 	msi_vecs = min(nvec, ZPCI_MSI_VEC_MAX);
 	msi_vecs = min_t(unsigned int, msi_vecs, CONFIG_PCI_NR_MSI);
 
arch/x86/kernel/apic/io_apic.c +1 -1
index e63a5bd..6126eaf 100644
@@ -3145,7 +3145,7 @@ int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 
 	/* Multiple MSI vectors only supported with interrupt remapping */
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
-		return 1;
+		return -EINVAL;
 
 	node = dev_to_node(&dev->dev);
 	irq_want = nr_irqs_gsi;
drivers/pci/msi.c +14 -38

101 lines not on this page · raw message

Signature · 6 lines
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#21 lore raw

RFC PATCH 8/77 PCI/MSI: Get rid of pci_enable_msi_block_auto() interface

In reply to Alexander Gordeev

As result of recent re-design of MSI/MSI-X interrupts enabling pattern pci_enable_msi_block_auto() interface became obsolete.

To enable maximum number of MSI interrupts for a device the driver will first obtain that number from pci_get_msi_cap() function and then call pci_enable_msi_block() interface.

---
 Documentation/PCI/MSI-HOWTO.txt |   30 ++----------------------------
 drivers/pci/msi.c               |   20 --------------------
 include/linux/pci.h             |    7 -------
 3 files changed, 2 insertions(+), 55 deletions(-)
Documentation/PCI/MSI-HOWTO.txt +2 -28
index 40abcfb..4d0525f 100644
@@ -133,33 +133,7 @@ static int foo_driver_enable_msi(struct foo_adapter *adapter, int nvec)
 	return rc;
 }
 
-4.2.3 pci_enable_msi_block_auto
-
-int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *count)
-
-This variation on pci_enable_msi() call allows a device driver to request
-the maximum possible number of MSIs.  The MSI specification only allows
-interrupts to be allocated in powers of two, up to a maximum of 2^5 (32).
-
-If this function returns a positive number, it indicates that it has
-succeeded and the returned value is the number of allocated interrupts. In
-this case, the function enables MSI on this device and updates dev->irq to
-be the lowest of the new interrupts assigned to it.  The other interrupts
-assigned to the device are in the range dev->irq to dev->irq + returned
-value - 1.
-
-If this function returns a negative number, it indicates an error and
-the driver should not attempt to request any more MSI interrupts for
-this device.
-
-If the device driver needs to know the number of interrupts the device
-supports it can pass the pointer count where that number is stored. The
-device driver must decide what action to take if pci_enable_msi_block_auto()
-succeeds, but returns a value less than the number of interrupts supported.
-If the device driver does not need to know the number of interrupts
-supported, it can set the pointer count to NULL.
-
-4.2.4 pci_disable_msi
+4.2.3 pci_disable_msi
 
 void pci_disable_msi(struct pci_dev *dev)
 
@@ -175,7 +149,7 @@ on any interrupt for which it previously called request_irq().
 Failure to do so results in a BUG_ON(), leaving the device with
 MSI enabled and thus leaking its vector.
 
-4.2.5 pci_get_msi_cap
+4.2.4 pci_get_msi_cap
 
 int pci_get_msi_cap(struct pci_dev *dev)
 
drivers/pci/msi.c +0 -20
index 583ace1..1934519 100644
@@ -849,26 +849,6 @@ int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec)
 }
 EXPORT_SYMBOL(pci_enable_msi_block);
 
-int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *maxvec)
-{
-	int ret, nvec;
-
-	ret = pci_get_msi_cap(dev);
-	if (ret < 0)
-		return ret;
-
-	if (maxvec)
-		*maxvec = ret;
-
-	nvec = ret;
-	ret = pci_enable_msi_block(dev, nvec);
-	if (ret)
-		return ret;
-
-	return nvec;
-}
-EXPORT_SYMBOL(pci_enable_msi_block_auto);
-
 void pci_msi_shutdown(struct pci_dev *dev)
 {
 	struct msi_desc *desc;
include/linux/pci.h +0 -7
index 2fa92ef..13bf88d 100644
@@ -1154,12 +1154,6 @@ static inline int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec)
 	return -1;
 }
 
-static inline int
-pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *maxvec)
-{
-	return -1;
-}
-
 static inline void pci_msi_shutdown(struct pci_dev *dev)
 { }
 static inline void pci_disable_msi(struct pci_dev *dev)
@@ -1192,7 +1186,6 @@ static inline int pci_msi_enabled(void)
 #else
 int pci_get_msi_cap(struct pci_dev *dev);
 int pci_enable_msi_block(struct pci_dev *dev, unsigned int nvec);
-int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *maxvec);
 void pci_msi_shutdown(struct pci_dev *dev);
 void pci_disable_msi(struct pci_dev *dev);
 int pci_msix_table_size(struct pci_dev *dev);
Signature · 1 line
1.7.7.6
#22 lore raw

RFC PATCH 9/77 ahci: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/ata/ahci.c |   56 ++++++++++++++++++++++++++++++++-------------------
 1 files changed, 35 insertions(+), 21 deletions(-)
drivers/ata/ahci.c +35 -21
index 9d715ae..f1c8838 100644
@@ -1091,26 +1091,40 @@ static inline void ahci_gtf_filter_workaround(struct ata_host *host)
 {}
 #endif
 
-int ahci_init_interrupts(struct pci_dev *pdev, struct ahci_host_priv *hpriv)
+int ahci_init_interrupts(struct pci_dev *pdev, unsigned int n_ports,
+			 struct ahci_host_priv *hpriv)
 {
-	int rc;
-	unsigned int maxvec;
+	int rc, nvec;
 
-	if (!(hpriv->flags & AHCI_HFLAG_NO_MSI)) {
-		rc = pci_enable_msi_block_auto(pdev, &maxvec);
-		if (rc > 0) {
-			if ((rc == maxvec) || (rc == 1))
-				return rc;
-			/*
-			 * Assume that advantage of multipe MSIs is negated,
-			 * so fallback to single MSI mode to save resources
-			 */
-			pci_disable_msi(pdev);
-			if (!pci_enable_msi(pdev))
-				return 1;
-		}
-	}
+	if (hpriv->flags & AHCI_HFLAG_NO_MSI)
+		goto intx;
+
+	rc = pci_get_msi_cap(pdev);
+	if (rc < 0)
+		goto intx;
+
+	/*
+	 * If number of MSIs is less than number of ports then Sharing Last
+	 * Message mode could be enforced. In this case assume that advantage
+	 * of multipe MSIs is negated and use single MSI mode instead.
+	 */
+	if (rc < n_ports)
+		goto single_msi;
+
+	nvec = rc;
+	rc = pci_enable_msi_block(pdev, nvec);
+	if (rc)
+		goto intx;
 
+	return nvec;
+
+single_msi:
+	rc = pci_enable_msi(pdev);
+	if (rc)
+		goto intx;
+	return 1;
+
+intx:
 	pci_intx(pdev, 1);
 	return 0;
 }
@@ -1277,10 +1291,6 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	hpriv->mmio = pcim_iomap_table(pdev)[ahci_pci_bar];
 
-	n_msis = ahci_init_interrupts(pdev, hpriv);
-	if (n_msis > 1)
-		hpriv->flags |= AHCI_HFLAG_MULTI_MSI;
-
 	/* save initial config */
 	ahci_pci_save_initial_config(pdev, hpriv);
 
@@ -1335,6 +1345,10 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	 */
 	n_ports = max(ahci_nr_ports(hpriv->cap), fls(hpriv->port_map));
 
+	n_msis = ahci_init_interrupts(pdev, n_ports, hpriv);
+	if (n_msis > 1)
+		hpriv->flags |= AHCI_HFLAG_MULTI_MSI;
+
 	host = ata_host_alloc_pinfo(&pdev->dev, ppi, n_ports);
 	if (!host)
 		return -ENOMEM;
Signature · 1 line
1.7.7.6
#23 lore raw

RFC PATCH 10/77 ahci: Check MRSM bit when multiple MSIs enabled

In reply to Alexander Gordeev

Do not trust the hardware and always check if MSI Revert to Single Message mode was enforced. Fall back to the single MSI mode in case it did. Not doing so might screw up the interrupt handling.

---
 drivers/ata/ahci.c |   17 +++++++++++++++++
 drivers/ata/ahci.h |    1 +
 2 files changed, 18 insertions(+), 0 deletions(-)
drivers/ata/ahci.c +17 -0
index f1c8838..3a39cc8 100644
@@ -1091,6 +1091,14 @@ static inline void ahci_gtf_filter_workaround(struct ata_host *host)
 {}
 #endif
 
+static int ahci_get_mrsm(struct ahci_host_priv *hpriv)
+{
+	void __iomem *mmio = hpriv->mmio;
+	u32 ctl = readl(mmio + HOST_CTL);
+
+	return ctl & HOST_MRSM;
+}
+
 int ahci_init_interrupts(struct pci_dev *pdev, unsigned int n_ports,
 			 struct ahci_host_priv *hpriv)
 {
@@ -1116,6 +1124,15 @@ int ahci_init_interrupts(struct pci_dev *pdev, unsigned int n_ports,
 	if (rc)
 		goto intx;
 
+	/*
+	 * Fallback to single MSI mode if the controller enforced MRSM mode 
+	 */
+	if (ahci_get_mrsm(hpriv)) {
+		pci_disable_msi(pdev);
+		printk(KERN_INFO "ahci: MRSM is on, fallback to single MSI\n");
+		goto single_msi;
+	}
+
 	return nvec;
 
 single_msi:
drivers/ata/ahci.h +1 -0
index 1145637..19bc846 100644
@@ -91,6 +91,7 @@ enum {
 	/* HOST_CTL bits */
 	HOST_RESET		= (1 << 0),  /* reset controller; self-clear */
 	HOST_IRQ_EN		= (1 << 1),  /* global IRQ enable */
+	HOST_MRSM		= (1 << 2),  /* MSI Revert to Single Message */
 	HOST_AHCI_EN		= (1 << 31), /* AHCI enabled */
 
 	/* HOST_CAP bits */
Signature · 1 line
1.7.7.6
#24 lore raw

RFC PATCH 11/77 benet: Return -ENOSPC when not enough MSI-Xs available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/emulex/benet/be_main.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/net/ethernet/emulex/benet/be_main.c +2 -0
index 100b528..3e2c834 100644
@@ -2378,6 +2378,8 @@ static int be_msix_enable(struct be_adapter *adapter)
 					 num_vec);
 		if (!status)
 			goto done;
+	} else (status > 0) {
+		status = -ENOSPC;
 	}
 
 	dev_warn(dev, "MSIx enable failed\n");
Signature · 1 line
1.7.7.6
#25 lore raw

RFC PATCH 12/77 benet: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/emulex/benet/be_main.c |   38 ++++++++++++++------------
 1 files changed, 20 insertions(+), 18 deletions(-)
drivers/net/ethernet/emulex/benet/be_main.c +20 -18
index 3e2c834..84d560d 100644
@@ -2366,29 +2366,23 @@ static int be_msix_enable(struct be_adapter *adapter)
 	else
 		num_vec = adapter->cfg_num_qs;
 
-	for (i = 0; i < num_vec; i++)
-		adapter->msix_entries[i].entry = i;
+	status = pci_msix_table_size(adapter->pdev);
+	if (status < 0)
+		goto fail;
 
-	status = pci_enable_msix(adapter->pdev, adapter->msix_entries, num_vec);
-	if (status == 0) {
-		goto done;
-	} else if (status >= MIN_MSIX_VECTORS) {
-		num_vec = status;
-		status = pci_enable_msix(adapter->pdev, adapter->msix_entries,
-					 num_vec);
-		if (!status)
-			goto done;
-	} else (status > 0) {
+	num_vec = min(num_vec, status);
+	if (num_vec < MIN_MSIX_VECTORS) {
 		status = -ENOSPC;
+		goto fail;
 	}
 
-	dev_warn(dev, "MSIx enable failed\n");
+	for (i = 0; i < num_vec; i++)
+		adapter->msix_entries[i].entry = i;
+
+	status = pci_enable_msix(adapter->pdev, adapter->msix_entries, num_vec);
+	if (status)
+		goto fail;
 
-	/* INTx is not supported in VFs, so fail probe if enable_msix fails */
-	if (!be_physfn(adapter))
-		return status;
-	return 0;
-done:
 	if (be_roce_supported(adapter) && num_vec > MIN_MSIX_VECTORS) {
 		adapter->num_msix_roce_vec = num_vec / 2;
 		dev_info(dev, "enabled %d MSI-x vector(s) for RoCE\n",
@@ -2400,6 +2394,14 @@ done:
 	dev_info(dev, "enabled %d MSI-x vector(s) for NIC\n",
 		 adapter->num_msix_vec);
 	return 0;
+
+fail:
+	dev_warn(dev, "MSIx enable failed\n");
+
+	/* INTx is not supported in VFs, so fail probe if enable_msix fails */
+	if (!be_physfn(adapter))
+		return status;
+	return 0;
 }
 
 static inline int be_msix_vec_get(struct be_adapter *adapter,
Signature · 1 line
1.7.7.6
#26 lore raw

RFC PATCH 13/77 bna: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/brocade/bna/bnad.c |   34 ++++++++++++------------------
 1 files changed, 14 insertions(+), 20 deletions(-)
drivers/net/ethernet/brocade/bna/bnad.c +14 -20
index b78e69e..d41257c 100644
@@ -2469,21 +2469,11 @@ bnad_enable_msix(struct bnad *bnad)
 	if (bnad->msix_table)
 		return;
 
-	bnad->msix_table =
-		kcalloc(bnad->msix_num, sizeof(struct msix_entry), GFP_KERNEL);
-
-	if (!bnad->msix_table)
+	ret = pci_msix_table_size(bnad->pcidev);
+	if (ret < 0)
 		goto intx_mode;
 
-	for (i = 0; i < bnad->msix_num; i++)
-		bnad->msix_table[i].entry = i;
-
-	ret = pci_enable_msix(bnad->pcidev, bnad->msix_table, bnad->msix_num);
-	if (ret > 0) {
-		/* Not enough MSI-X vectors. */
-		pr_warn("BNA: %d MSI-X vectors allocated < %d requested\n",
-			ret, bnad->msix_num);
-
+	if (ret < bnad->msix_num) {
 		spin_lock_irqsave(&bnad->bna_lock, flags);
 		/* ret = #of vectors that we got */
 		bnad_q_num_adjust(bnad, (ret - BNAD_MAILBOX_MSIX_VECTORS) / 2,
@@ -2495,15 +2485,19 @@ bnad_enable_msix(struct bnad *bnad)
 
 		if (bnad->msix_num > ret)
 			goto intx_mode;
+	}
 
-		/* Try once more with adjusted numbers */
-		/* If this fails, fall back to INTx */
-		ret = pci_enable_msix(bnad->pcidev, bnad->msix_table,
-				      bnad->msix_num);
-		if (ret)
-			goto intx_mode;
+	bnad->msix_table =
+		kcalloc(bnad->msix_num, sizeof(struct msix_entry), GFP_KERNEL);
+
+	if (!bnad->msix_table)
+		goto intx_mode;
 
-	} else if (ret < 0)
+	for (i = 0; i < bnad->msix_num; i++)
+		bnad->msix_table[i].entry = i;
+
+	ret = pci_enable_msix(bnad->pcidev, bnad->msix_table, bnad->msix_num);
+	if (ret)
 		goto intx_mode;
 
 	pci_intx(bnad->pcidev, 0);
Signature · 1 line
1.7.7.6
#27 lore raw

RFC PATCH 14/77 bnx2x: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c |   54 ++++++++++-------------
 1 files changed, 23 insertions(+), 31 deletions(-)
drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +23 -31
index 61726af..edf31d2 100644
@@ -1564,7 +1564,7 @@ void bnx2x_free_irq(struct bnx2x *bp)
 
 int bnx2x_enable_msix(struct bnx2x *bp)
 {
-	int msix_vec = 0, i, rc;
+	int msix_vec = 0, nvec, i, rc;
 
 	/* VFs don't have a default status block */
 	if (IS_PF(bp)) {
@@ -1590,60 +1590,52 @@ int bnx2x_enable_msix(struct bnx2x *bp)
 		msix_vec++;
 	}
 
+	rc = pci_msix_table_size(bp->pdev);
+	if (rc < 0)
+		goto no_msix;
+
+	nvec = min(msix_vec, rc);
+	if (nvec < BNX2X_MIN_MSIX_VEC_CNT(bp))
+		nvec = 1;
+
 	DP(BNX2X_MSG_SP, "about to request enable msix with %d vectors\n",
 	   msix_vec);
 
-	rc = pci_enable_msix(bp->pdev, &bp->msix_table[0], msix_vec);
+	rc = pci_enable_msix(bp->pdev, &bp->msix_table[0], nvec);
+	if (rc)
+		goto no_msix;
 
 	/*
 	 * reconfigure number of tx/rx queues according to available
 	 * MSI-X vectors
 	 */
-	if (rc >= BNX2X_MIN_MSIX_VEC_CNT(bp)) {
-		/* how less vectors we will have? */
-		int diff = msix_vec - rc;
-
-		BNX2X_DEV_INFO("Trying to use less MSI-X vectors: %d\n", rc);
+	if (nvec == 1) {
+		bp->flags |= USING_SINGLE_MSIX_FLAG;
 
-		rc = pci_enable_msix(bp->pdev, &bp->msix_table[0], rc);
+		bp->num_ethernet_queues = 1;
+		bp->num_queues = bp->num_ethernet_queues + bp->num_cnic_queues;
+	} else if (nvec < msix_vec) {
+		/* how less vectors we will have? */
+		int diff = msix_vec - nvec;
 
-		if (rc) {
-			BNX2X_DEV_INFO("MSI-X is not attainable rc %d\n", rc);
-			goto no_msix;
-		}
 		/*
 		 * decrease number of queues by number of unallocated entries
 		 */
 		bp->num_ethernet_queues -= diff;
 		bp->num_queues = bp->num_ethernet_queues + bp->num_cnic_queues;
+	}
 
+	if (nvec != msix_vec)
 		BNX2X_DEV_INFO("New queue configuration set: %d\n",
 			       bp->num_queues);
-	} else if (rc > 0) {
-		/* Get by with single vector */
-		rc = pci_enable_msix(bp->pdev, &bp->msix_table[0], 1);
-		if (rc) {
-			BNX2X_DEV_INFO("Single MSI-X is not attainable rc %d\n",
-				       rc);
-			goto no_msix;
-		}
-
-		BNX2X_DEV_INFO("Using single MSI-X vector\n");
-		bp->flags |= USING_SINGLE_MSIX_FLAG;
-
-		BNX2X_DEV_INFO("set number of queues to 1\n");
-		bp->num_ethernet_queues = 1;
-		bp->num_queues = bp->num_ethernet_queues + bp->num_cnic_queues;
-	} else if (rc < 0) {
-		BNX2X_DEV_INFO("MSI-X is not attainable  rc %d\n", rc);
-		goto no_msix;
-	}
 
 	bp->flags |= USING_MSIX_FLAG;
 
 	return 0;
 
 no_msix:
+	BNX2X_DEV_INFO("MSI-X is not attainable rc %d\n", rc);
+
 	/* fall to INTx if not enough memory */
 	if (rc == -ENOMEM)
 		bp->flags |= DISABLE_MSI_FLAG;
Signature · 1 line
1.7.7.6
#28 lore raw

RFC PATCH 15/77 bnx2: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/broadcom/bnx2.c |   27 ++++++++++++++-------------
 1 files changed, 14 insertions(+), 13 deletions(-)
drivers/net/ethernet/broadcom/bnx2.c +14 -13
index e838a3f..c902627 100644
@@ -6202,25 +6202,26 @@ bnx2_enable_msix(struct bnx2 *bp, int msix_vecs)
 	 *  is setup properly */
 	BNX2_RD(bp, BNX2_PCI_MSIX_CONTROL);
 
-	for (i = 0; i < BNX2_MAX_MSIX_VEC; i++) {
-		msix_ent[i].entry = i;
-		msix_ent[i].vector = 0;
-	}
-
 	total_vecs = msix_vecs;
 #ifdef BCM_CNIC
 	total_vecs++;
 #endif
-	rc = -ENOSPC;
-	while (total_vecs >= BNX2_MIN_MSIX_VEC) {
-		rc = pci_enable_msix(bp->pdev, msix_ent, total_vecs);
-		if (rc <= 0)
-			break;
-		if (rc > 0)
-			total_vecs = rc;
+	rc = pci_msix_table_size(bp->pdev);
+	if (rc < 0)
+		return;
+
+	total_vecs = min(total_vecs, rc);
+	if (total_vecs < BNX2_MIN_MSIX_VEC)
+		return;
+
+	BUG_ON(total_vecs > ARRAY_SIZE(msix_ent));
+	for (i = 0; i < total_vecs; i++) {
+		msix_ent[i].entry = i;
+		msix_ent[i].vector = 0;
 	}
 
-	if (rc != 0)
+	rc = pci_enable_msix(bp->pdev, msix_ent, total_vecs);
+	if (rc)
 		return;
 
 	msix_vecs = total_vecs;
Signature · 1 line
1.7.7.6
#29 lore raw

RFC PATCH 16/77 cciss: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/block/cciss.c |   17 +++++++----------
 1 files changed, 7 insertions(+), 10 deletions(-)
drivers/block/cciss.c +7 -10
index d2d95ff..80068a0 100644
@@ -4079,7 +4079,11 @@ static void cciss_interrupt_mode(ctlr_info_t *h)
 		goto default_int_mode;
 
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSIX)) {
-		err = pci_enable_msix(h->pdev, cciss_msix_entries, 4);
+		err = pci_msix_table_size(h->pdev);
+		if (err < ARRAY_SIZE(cciss_msix_entries))
+			goto default_int_mode;
+		err = pci_enable_msix(h->pdev, cciss_msix_entries,
+				      ARRAY_SIZE(cciss_msix_entries));
 		if (!err) {
 			h->intr[0] = cciss_msix_entries[0].vector;
 			h->intr[1] = cciss_msix_entries[1].vector;
@@ -4088,15 +4092,8 @@ static void cciss_interrupt_mode(ctlr_info_t *h)
 			h->msix_vector = 1;
 			return;
 		}
-		if (err > 0) {
-			dev_warn(&h->pdev->dev,
-				"only %d MSI-X vectors available\n", err);
-			goto default_int_mode;
-		} else {
-			dev_warn(&h->pdev->dev,
-				"MSI-X init failed %d\n", err);
-			goto default_int_mode;
-		}
+		dev_warn(&h->pdev->dev, "MSI-X init failed %d\n", err);
+		goto default_int_mode;
 	}
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSI)) {
 		if (!pci_enable_msi(h->pdev))
Signature · 1 line
1.7.7.6
#30 lore raw

RFC PATCH 17/77 cciss: Update a misleading comment on interrupt usage

In reply to Alexander Gordeev
---
 drivers/block/cciss.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
drivers/block/cciss.c +4 -1
index 80068a0..bf11540 100644
@@ -4103,7 +4103,10 @@ static void cciss_interrupt_mode(ctlr_info_t *h)
 	}
 default_int_mode:
 #endif				/* CONFIG_PCI_MSI */
-	/* if we get here we're going to use the default interrupt mode */
+	/*
+	 * If we get here we're going to use either the
+	 * default interrupt mode or single MSI mode
+	 */
 	h->intr[h->intr_mode] = h->pdev->irq;
 	return;
 }
Signature · 1 line
1.7.7.6
#31 lore raw

RFC PATCH 18/77 cciss: Fallback to single MSI mode in case MSI-X failed

In reply to Alexander Gordeev
---
 drivers/block/cciss.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/block/cciss.c +2 -2
index bf11540..0eea035 100644
@@ -4081,7 +4081,7 @@ static void cciss_interrupt_mode(ctlr_info_t *h)
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSIX)) {
 		err = pci_msix_table_size(h->pdev);
 		if (err < ARRAY_SIZE(cciss_msix_entries))
-			goto default_int_mode;
+			goto single_msi_mode;
 		err = pci_enable_msix(h->pdev, cciss_msix_entries,
 				      ARRAY_SIZE(cciss_msix_entries));
 		if (!err) {
@@ -4093,8 +4093,8 @@ static void cciss_interrupt_mode(ctlr_info_t *h)
 			return;
 		}
 		dev_warn(&h->pdev->dev, "MSI-X init failed %d\n", err);
-		goto default_int_mode;
 	}
+single_msi_mode:
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSI)) {
 		if (!pci_enable_msi(h->pdev))
 			h->msi_vector = 1;
Signature · 1 line
1.7.7.6
#32 lore raw

RFC PATCH 19/77 csiostor: Do not call pci_disable_msix() if pci_enable_msix() failed

In reply to Alexander Gordeev
---
 drivers/scsi/csiostor/csio_isr.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)
drivers/scsi/csiostor/csio_isr.c +1 -3
index 7ee9777..91ba91d 100644
@@ -529,10 +529,8 @@ csio_enable_msix(struct csio_hw *hw)
 			csio_reduce_sqsets(hw, cnt - extra);
 		}
 	} else {
-		if (rv > 0) {
-			pci_disable_msix(hw->pdev);
+		if (rv > 0)
 			csio_info(hw, "Not using MSI-X, remainder:%d\n", rv);
-		}
 
 		kfree(entries);
 		return -ENOMEM;
Signature · 1 line
1.7.7.6
#33 lore raw

RFC PATCH 20/77 csiostor: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/scsi/csiostor/csio_isr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/scsi/csiostor/csio_isr.c +1 -1
index 91ba91d..abf20ab 100644
@@ -533,7 +533,7 @@ csio_enable_msix(struct csio_hw *hw)
 			csio_info(hw, "Not using MSI-X, remainder:%d\n", rv);
 
 		kfree(entries);
-		return -ENOMEM;
+		return -ENOSPC;
 	}
 
 	/* Save off vectors */
Signature · 1 line
1.7.7.6
#34 lore raw

RFC PATCH 21/77 csiostor: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/csiostor/csio_isr.c |   18 ++++++++++++------
 1 files changed, 12 insertions(+), 6 deletions(-)
drivers/scsi/csiostor/csio_isr.c +12 -6
index abf20ab..b8a840e 100644
@@ -512,6 +512,16 @@ csio_enable_msix(struct csio_hw *hw)
 	if (hw->flags & CSIO_HWF_USING_SOFT_PARAMS || !csio_is_hw_master(hw))
 		cnt = min_t(uint8_t, hw->cfg_niq, cnt);
 
+	rv = pci_msix_table_size(hw->pdev);
+	if (rv < 0)
+		return rv;
+
+	cnt = min(cnt, rv);
+	if (cnt < min) {
+		csio_info(hw, "Not using MSI-X, remainder:%d\n", rv);
+		return -ENOSPC;
+	}
+
 	entries = kzalloc(sizeof(struct msix_entry) * cnt, GFP_KERNEL);
 	if (!entries)
 		return -ENOMEM;
@@ -521,19 +531,15 @@ csio_enable_msix(struct csio_hw *hw)
 
 	csio_dbg(hw, "FW supp #niq:%d, trying %d msix's\n", hw->cfg_niq, cnt);
 
-	while ((rv = pci_enable_msix(hw->pdev, entries, cnt)) >= min)
-		cnt = rv;
+	rv = pci_enable_msix(hw->pdev, entries, cnt);
 	if (!rv) {
 		if (cnt < (hw->num_sqsets + extra)) {
 			csio_dbg(hw, "Reducing sqsets to %d\n", cnt - extra);
 			csio_reduce_sqsets(hw, cnt - extra);
 		}
 	} else {
-		if (rv > 0)
-			csio_info(hw, "Not using MSI-X, remainder:%d\n", rv);
-
 		kfree(entries);
-		return -ENOSPC;
+		return rv;
 	}
 
 	/* Save off vectors */
Signature · 1 line
1.7.7.6
#35 lore raw

RFC PATCH 22/77 cxgb3: Do not call pci_disable_msix() if pci_enable_msix() failed

In reply to Alexander Gordeev
---
 drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c +0 -3
index b650951..9bd3099 100644
@@ -3097,9 +3097,6 @@ static int cxgb_enable_msix(struct adapter *adap)
 	while ((err = pci_enable_msix(adap->pdev, entries, vectors)) > 0)
 		vectors = err;
 
-	if (err < 0)
-		pci_disable_msix(adap->pdev);
-
 	if (!err && vectors < (adap->params.nports + 1)) {
 		pci_disable_msix(adap->pdev);
 		err = -1;
Signature · 1 line
1.7.7.6
#36 lore raw

RFC PATCH 23/77 cxgb3: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c +1 -1
index 9bd3099..915729c 100644
@@ -3099,7 +3099,7 @@ static int cxgb_enable_msix(struct adapter *adap)
 
 	if (!err && vectors < (adap->params.nports + 1)) {
 		pci_disable_msix(adap->pdev);
-		err = -1;
+		err = -ENOSPC;
 	}
 
 	if (!err) {
Signature · 1 line
1.7.7.6
#37 lore raw

RFC PATCH 24/77 cxgb3: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c |   29 ++++++++++++-----------
 1 files changed, 15 insertions(+), 14 deletions(-)
drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c +15 -14
index 915729c..bf14fd6 100644
@@ -3090,25 +3090,26 @@ static int cxgb_enable_msix(struct adapter *adap)
 	int vectors;
 	int i, err;
 
-	vectors = ARRAY_SIZE(entries);
+	err = pci_msix_table_size(adap->pdev);
+	if (err < 0)
+		return err;
+
+	vectors = min_t(int, err, ARRAY_SIZE(entries));
+	if (vectors < (adap->params.nports + 1))
+		return -ENOSPC;
+
 	for (i = 0; i < vectors; ++i)
 		entries[i].entry = i;
 
-	while ((err = pci_enable_msix(adap->pdev, entries, vectors)) > 0)
-		vectors = err;
-
-	if (!err && vectors < (adap->params.nports + 1)) {
-		pci_disable_msix(adap->pdev);
-		err = -ENOSPC;
-	}
+	err = pci_enable_msix(adap->pdev, entries, vectors);
+	if (err)
+		return err;
 
-	if (!err) {
-		for (i = 0; i < vectors; ++i)
-			adap->msix_info[i].vec = entries[i].vector;
-		adap->msix_nvectors = vectors;
-	}
+	for (i = 0; i < vectors; ++i)
+		adap->msix_info[i].vec = entries[i].vector;
+	adap->msix_nvectors = vectors;
 
-	return err;
+	return 0;
 }
 
 static void print_port_info(struct adapter *adap, const struct adapter_info *ai)
Signature · 1 line
1.7.7.6
#38 lore raw

RFC PATCH 25/77 cxgb4: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +3 -1
index c73cabd..9425bc6 100644
@@ -5732,9 +5732,11 @@ static int enable_msix(struct adapter *adap)
 		}
 		for (i = 0; i < want; ++i)
 			adap->msix_info[i].vec = entries[i].vector;
-	} else if (err > 0)
+	} else if (err > 0) {
 		dev_info(adap->pdev_dev,
 			 "only %d MSI-X vectors left, not using MSI-X\n", err);
+		err = -ENOSPC;
+	}
 	return err;
 }
 
Signature · 1 line
1.7.7.6
#39 lore raw

RFC PATCH 26/77 cxgb4: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c |   62 +++++++++++++----------
 1 files changed, 35 insertions(+), 27 deletions(-)
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +35 -27
index 9425bc6..e5fbbd3 100644
@@ -5699,9 +5699,6 @@ static int enable_msix(struct adapter *adap)
 	unsigned int nchan = adap->params.nports;
 	struct msix_entry entries[MAX_INGQ + 1];
 
-	for (i = 0; i < ARRAY_SIZE(entries); ++i)
-		entries[i].entry = i;
-
 	want = s->max_ethqsets + EXTRA_VECS;
 	if (is_offload(adap)) {
 		want += s->rdmaqs + s->ofldqsets;
@@ -5710,34 +5707,45 @@ static int enable_msix(struct adapter *adap)
 	}
 	need = adap->params.nports + EXTRA_VECS + ofld_need;
 
-	while ((err = pci_enable_msix(adap->pdev, entries, want)) >= need)
-		want = err;
+	err = pci_msix_table_size(adap->pdev);
+	if (err < 0)
+		return err;
 
-	if (!err) {
-		/*
-		 * Distribute available vectors to the various queue groups.
-		 * Every group gets its minimum requirement and NIC gets top
-		 * priority for leftovers.
-		 */
-		i = want - EXTRA_VECS - ofld_need;
-		if (i < s->max_ethqsets) {
-			s->max_ethqsets = i;
-			if (i < s->ethqsets)
-				reduce_ethqs(adap, i);
-		}
-		if (is_offload(adap)) {
-			i = want - EXTRA_VECS - s->max_ethqsets;
-			i -= ofld_need - nchan;
-			s->ofldqsets = (i / nchan) * nchan;  /* round down */
-		}
-		for (i = 0; i < want; ++i)
-			adap->msix_info[i].vec = entries[i].vector;
-	} else if (err > 0) {
+	want = min(want, err);
+	if (want < need) {
 		dev_info(adap->pdev_dev,
 			 "only %d MSI-X vectors left, not using MSI-X\n", err);
-		err = -ENOSPC;
+		return -ENOSPC;
 	}
-	return err;
+
+	BUG_ON(want > ARRAY_SIZE(entries));
+	for (i = 0; i < want; ++i)
+		entries[i].entry = i;
+
+	err = pci_enable_msix(adap->pdev, entries, want);
+	if (err)
+		return err;
+
+	/*
+	 * Distribute available vectors to the various queue groups.
+	 * Every group gets its minimum requirement and NIC gets top
+	 * priority for leftovers.
+	 */
+	i = want - EXTRA_VECS - ofld_need;
+	if (i < s->max_ethqsets) {
+		s->max_ethqsets = i;
+		if (i < s->ethqsets)
+			reduce_ethqs(adap, i);
+	}
+	if (is_offload(adap)) {
+		i = want - EXTRA_VECS - s->max_ethqsets;
+		i -= ofld_need - nchan;
+		s->ofldqsets = (i / nchan) * nchan;  /* round down */
+	}
+	for (i = 0; i < want; ++i)
+		adap->msix_info[i].vec = entries[i].vector;
+
+	return 0;
 }
 
 #undef EXTRA_VECS
Signature · 1 line
1.7.7.6
#40 lore raw

RFC PATCH 27/77 cxgb4vf: Do not call pci_disable_msix() if pci_enable_msix() failed

In reply to Alexander Gordeev
---
 .../net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c    |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +0 -1
index 40c22e7..87a82fc 100644
@@ -2468,7 +2468,6 @@ static int enable_msix(struct adapter *adapter)
 		for (i = 0; i < want; ++i)
 			adapter->msix_info[i].vec = entries[i].vector;
 	} else if (err > 0) {
-		pci_disable_msix(adapter->pdev);
 		dev_info(adapter->pdev_dev, "only %d MSI-X vectors left,"
 			 " not using MSI-X\n", err);
 	}
Signature · 1 line
1.7.7.6
#41 lore raw

RFC PATCH 28/77 cxgb4vf: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 .../net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c    |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +1 -0
index 87a82fc..11cbce1 100644
@@ -2470,6 +2470,7 @@ static int enable_msix(struct adapter *adapter)
 	} else if (err > 0) {
 		dev_info(adapter->pdev_dev, "only %d MSI-X vectors left,"
 			 " not using MSI-X\n", err);
+		err = -ENOSPC;
 	}
 	return err;
 }
Signature · 1 line
1.7.7.6
#42 lore raw

RFC PATCH 29/77 cxgb4vf: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 .../net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c    |   49 ++++++++++++--------
 1 files changed, 29 insertions(+), 20 deletions(-)
drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +29 -20
index 11cbce1..48bb33b 100644
@@ -2437,13 +2437,10 @@ static void reduce_ethqs(struct adapter *adapter, int n)
  */
 static int enable_msix(struct adapter *adapter)
 {
-	int i, err, want, need;
+	int i, err, want, need, nqsets;
 	struct msix_entry entries[MSIX_ENTRIES];
 	struct sge *s = &adapter->sge;
 
-	for (i = 0; i < MSIX_ENTRIES; ++i)
-		entries[i].entry = i;
-
 	/*
 	 * We _want_ enough MSI-X interrupts to cover all of our "Queue Sets"
 	 * plus those needed for our "extras" (for example, the firmware
@@ -2453,26 +2450,38 @@ static int enable_msix(struct adapter *adapter)
 	 */
 	want = s->max_ethqsets + MSIX_EXTRAS;
 	need = adapter->params.nports + MSIX_EXTRAS;
-	while ((err = pci_enable_msix(adapter->pdev, entries, want)) >= need)
-		want = err;
 
-	if (err == 0) {
-		int nqsets = want - MSIX_EXTRAS;
-		if (nqsets < s->max_ethqsets) {
-			dev_warn(adapter->pdev_dev, "only enough MSI-X vectors"
-				 " for %d Queue Sets\n", nqsets);
-			s->max_ethqsets = nqsets;
-			if (nqsets < s->ethqsets)
-				reduce_ethqs(adapter, nqsets);
-		}
-		for (i = 0; i < want; ++i)
-			adapter->msix_info[i].vec = entries[i].vector;
-	} else if (err > 0) {
+	err = pci_msix_table_size(adapter->pdev);
+	if (err < 0)
+		return err;
+
+	want = min(want, err);
+	if (want < need) {
 		dev_info(adapter->pdev_dev, "only %d MSI-X vectors left,"
 			 " not using MSI-X\n", err);
-		err = -ENOSPC;
+		return -ENOSPC;
 	}
-	return err;
+
+	BUG_ON(want > ARRAY_SIZE(entries));
+	for (i = 0; i < want; ++i)
+		entries[i].entry = i;
+
+	err = pci_enable_msix(adapter->pdev, entries, want);
+	if (err)
+		return err;
+
+	nqsets = want - MSIX_EXTRAS;
+	if (nqsets < s->max_ethqsets) {
+		dev_warn(adapter->pdev_dev, "only enough MSI-X vectors"
+			 " for %d Queue Sets\n", nqsets);
+		s->max_ethqsets = nqsets;
+		if (nqsets < s->ethqsets)
+			reduce_ethqs(adapter, nqsets);
+	}
+	for (i = 0; i < want; ++i)
+		adapter->msix_info[i].vec = entries[i].vector;
+
+	return 0;
 }
 
 static const struct net_device_ops cxgb4vf_netdev_ops	= {
Signature · 1 line
1.7.7.6
#43 lore raw

RFC PATCH 30/77 hpsa: Update a misleading comment on interrupt usage

In reply to Alexander Gordeev
---
 drivers/scsi/hpsa.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)
drivers/scsi/hpsa.c +5 -1
index 891c86b..393c8db 100644
@@ -4141,7 +4141,11 @@ static void hpsa_interrupt_mode(struct ctlr_info *h)
 	}
 default_int_mode:
 #endif				/* CONFIG_PCI_MSI */
-	/* if we get here we're going to use the default interrupt mode */
+	/*
+	 * If we get here we're going to use either the
+	 * default interrupt mode or single MSI mode
+	 */
+
 	h->intr[h->intr_mode] = h->pdev->irq;
 }
 
Signature · 1 line
1.7.7.6
#44 lore raw

RFC PATCH 31/77 hpsa: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/hpsa.c |   28 +++++++++++++---------------
 1 files changed, 13 insertions(+), 15 deletions(-)
drivers/scsi/hpsa.c +13 -15
index 393c8db..eb17b3d 100644
@@ -4103,34 +4103,32 @@ static void hpsa_interrupt_mode(struct ctlr_info *h)
 	int err, i;
 	struct msix_entry hpsa_msix_entries[MAX_REPLY_QUEUES];
 
-	for (i = 0; i < MAX_REPLY_QUEUES; i++) {
-		hpsa_msix_entries[i].vector = 0;
-		hpsa_msix_entries[i].entry = i;
-	}
-
 	/* Some boards advertise MSI but don't really support it */
 	if ((h->board_id == 0x40700E11) || (h->board_id == 0x40800E11) ||
 	    (h->board_id == 0x40820E11) || (h->board_id == 0x40830E11))
 		goto default_int_mode;
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSIX)) {
 		dev_info(&h->pdev->dev, "MSIX\n");
+
+		err = pci_msix_table_size(h->pdev);
+		if (err < ARRAY_SIZE(hpsa_msix_entries))
+			goto default_int_mode;
+
+		for (i = 0; i < ARRAY_SIZE(hpsa_msix_entries); i++) {
+			hpsa_msix_entries[i].vector = 0;
+			hpsa_msix_entries[i].entry = i;
+		}
+
 		err = pci_enable_msix(h->pdev, hpsa_msix_entries,
-						MAX_REPLY_QUEUES);
+				      ARRAY_SIZE(hpsa_msix_entries));
 		if (!err) {
 			for (i = 0; i < MAX_REPLY_QUEUES; i++)
 				h->intr[i] = hpsa_msix_entries[i].vector;
 			h->msix_vector = 1;
 			return;
 		}
-		if (err > 0) {
-			dev_warn(&h->pdev->dev, "only %d MSI-X vectors "
-			       "available\n", err);
-			goto default_int_mode;
-		} else {
-			dev_warn(&h->pdev->dev, "MSI-X init failed %d\n",
-			       err);
-			goto default_int_mode;
-		}
+		dev_warn(&h->pdev->dev, "MSI-X init failed %d\n", err);
+		goto default_int_mode;
 	}
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSI)) {
 		dev_info(&h->pdev->dev, "MSI\n");
Signature · 1 line
1.7.7.6
#45 lore raw

RFC PATCH 32/77 hpsa: Fallback to single MSI mode in case MSI-X failed

In reply to Alexander Gordeev
---
 drivers/scsi/hpsa.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
drivers/scsi/hpsa.c +3 -2
index eb17b3d..252b65d 100644
@@ -4112,7 +4112,7 @@ static void hpsa_interrupt_mode(struct ctlr_info *h)
 
 		err = pci_msix_table_size(h->pdev);
 		if (err < ARRAY_SIZE(hpsa_msix_entries))
-			goto default_int_mode;
+			goto single_msi_mode;
 
 		for (i = 0; i < ARRAY_SIZE(hpsa_msix_entries); i++) {
 			hpsa_msix_entries[i].vector = 0;
@@ -4128,8 +4128,9 @@ static void hpsa_interrupt_mode(struct ctlr_info *h)
 			return;
 		}
 		dev_warn(&h->pdev->dev, "MSI-X init failed %d\n", err);
-		goto default_int_mode;
+		goto single_msi_mode;
 	}
+single_msi_mode:
 	if (pci_find_capability(h->pdev, PCI_CAP_ID_MSI)) {
 		dev_info(&h->pdev->dev, "MSI\n");
 		if (!pci_enable_msi(h->pdev))
Signature · 1 line
1.7.7.6
#46 lore raw

RFC PATCH 33/77 ioat: Disable MSI-X in case request of IRQ failed

In reply to Alexander Gordeev
---
 drivers/dma/ioat/dma.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/dma/ioat/dma.c +1 -0
index 5ff6fc1..acee3b2 100644
@@ -937,6 +937,7 @@ msix:
 				chan = ioat_chan_by_index(device, j);
 				devm_free_irq(dev, msix->vector, chan);
 			}
+			pci_disable_msix(pdev);
 			goto msix_single_vector;
 		}
 	}
Signature · 1 line
1.7.7.6
#47 lore raw

RFC PATCH 34/77 ioat: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/dma/ioat/dma.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
drivers/dma/ioat/dma.c +7 -3
index acee3b2..b352ee6 100644
@@ -915,15 +915,19 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device)
 
 msix:
 	/* The number of MSI-X vectors should equal the number of channels */
+	err = pci_msix_table_size(pdev);
+	if (err < 0)
+		goto msi;
+	if (err < device->common.chancnt)
+		goto msix_single_vector;
+
 	msixcnt = device->common.chancnt;
 	for (i = 0; i < msixcnt; i++)
 		device->msix_entries[i].entry = i;
 
 	err = pci_enable_msix(pdev, device->msix_entries, msixcnt);
-	if (err < 0)
+	if (err)
 		goto msi;
-	if (err > 0)
-		goto msix_single_vector;
 
 	for (i = 0; i < msixcnt; i++) {
 		msix = &device->msix_entries[i];
Signature · 1 line
1.7.7.6
#52 lore raw

RFC PATCH 35/77 ipr: Do not call pci_disable_msi/msix() if pci_enable_msi/msix() failed

In reply to Alexander Gordeev
---
 drivers/scsi/ipr.c |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)
drivers/scsi/ipr.c +2 -6
index 36ac1c3..fb57e21 100644
@@ -9255,10 +9255,8 @@ static int ipr_enable_msix(struct ipr_ioa_cfg *ioa_cfg)
 	while ((err = pci_enable_msix(ioa_cfg->pdev, entries, vectors)) > 0)
 			vectors = err;
 
-	if (err < 0) {
-		pci_disable_msix(ioa_cfg->pdev);
+	if (err < 0)
 		return err;
-	}
 
 	if (!err) {
 		for (i = 0; i < vectors; i++)
@@ -9278,10 +9276,8 @@ static int ipr_enable_msi(struct ipr_ioa_cfg *ioa_cfg)
 	while ((err = pci_enable_msi_block(ioa_cfg->pdev, vectors)) > 0)
 			vectors = err;
 
-	if (err < 0) {
-		pci_disable_msi(ioa_cfg->pdev);
+	if (err < 0)
 		return err;
-	}
 
 	if (!err) {
 		for (i = 0; i < vectors; i++)
Signature · 1 line
1.7.7.6
#53 lore raw

RFC PATCH 36/77 ipr: Enable MSI-X when IPR_USE_MSIX type is set, not IPR_USE_MSI

In reply to Alexander Gordeev
---
 drivers/scsi/ipr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/scsi/ipr.c +1 -1
index fb57e21..762a93e 100644
@@ -9527,7 +9527,7 @@ static int ipr_probe_ioa(struct pci_dev *pdev,
 		ipr_number_of_msix = IPR_MAX_MSIX_VECTORS;
 	}
 
-	if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSI &&
+	if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSIX &&
 			ipr_enable_msix(ioa_cfg) == 0)
 		ioa_cfg->intr_flag = IPR_USE_MSIX;
 	else if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSI &&
Signature · 1 line
1.7.7.6
#55 lore raw

RFC PATCH 37/77 ipr: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/ipr.c |   46 +++++++++++++++++++++++-----------------------
 1 files changed, 23 insertions(+), 23 deletions(-)
drivers/scsi/ipr.c +23 -23
index 762a93e..86ed0a2 100644
@@ -9247,45 +9247,45 @@ static int ipr_enable_msix(struct ipr_ioa_cfg *ioa_cfg)
 	struct msix_entry entries[IPR_MAX_MSIX_VECTORS];
 	int i, err, vectors;
 
-	for (i = 0; i < ARRAY_SIZE(entries); ++i)
-		entries[i].entry = i;
+	err = pci_msix_table_size(ioa_cfg->pdev);
+	if (err < 0)
+		return err;
 
-	vectors = ipr_number_of_msix;
+	vectors = min_t(int, err, ipr_number_of_msix);
 
-	while ((err = pci_enable_msix(ioa_cfg->pdev, entries, vectors)) > 0)
-			vectors = err;
+	BUG_ON(vectors > ARRAY_SIZE(entries));
+	for (i = 0; i < vectors; ++i)
+		entries[i].entry = i;
 
-	if (err < 0)
+	err = pci_enable_msix(ioa_cfg->pdev, entries, vectors);
+	if (err)
 		return err;
 
-	if (!err) {
-		for (i = 0; i < vectors; i++)
-			ioa_cfg->vectors_info[i].vec = entries[i].vector;
-		ioa_cfg->nvectors = vectors;
-	}
+	for (i = 0; i < vectors; i++)
+		ioa_cfg->vectors_info[i].vec = entries[i].vector;
+	ioa_cfg->nvectors = vectors;
 
-	return err;
+	return 0;
 }
 
 static int ipr_enable_msi(struct ipr_ioa_cfg *ioa_cfg)
 {
 	int i, err, vectors;
 
-	vectors = ipr_number_of_msix;
-
-	while ((err = pci_enable_msi_block(ioa_cfg->pdev, vectors)) > 0)
-			vectors = err;
-
+	err = pci_get_msi_cap(ioa_cfg->pdev);
 	if (err < 0)
 		return err;
 
-	if (!err) {
-		for (i = 0; i < vectors; i++)
-			ioa_cfg->vectors_info[i].vec = ioa_cfg->pdev->irq + i;
-		ioa_cfg->nvectors = vectors;
-	}
+	vectors = min_t(int, err, ipr_number_of_msix);
+	err = pci_enable_msi_block(ioa_cfg->pdev, vectors);
+	if (err)
+		return err;
+
+	for (i = 0; i < vectors; i++)
+		ioa_cfg->vectors_info[i].vec = ioa_cfg->pdev->irq + i;
+	ioa_cfg->nvectors = vectors;
 
-	return err;
+	return 0;
 }
 
 static void name_msi_vectors(struct ipr_ioa_cfg *ioa_cfg)
Signature · 1 line
1.7.7.6
#56 lore raw

RFC PATCH 38/77 ixgbe: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c |   62 +++++++++++++------------
 1 files changed, 32 insertions(+), 30 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +32 -30
index 90b4e10..2444a4d 100644
@@ -711,37 +711,39 @@ static void ixgbe_acquire_msix_vectors(struct ixgbe_adapter *adapter,
 	 * Right now, we simply care about how many we'll get; we'll
 	 * set them up later while requesting irq's.
 	 */
-	while (vectors >= vector_threshold) {
-		err = pci_enable_msix(adapter->pdev, adapter->msix_entries,
-				      vectors);
-		if (!err) /* Success in acquiring all requested vectors. */
-			break;
-		else if (err < 0)
-			vectors = 0; /* Nasty failure, quit now */
-		else /* err == number of vectors we should try again with */
-			vectors = err;
-	}
+	err = pci_msix_table_size(adapter->pdev);
+	if (err < 0)
+		goto err_alloc_msix;
 
-	if (vectors < vector_threshold) {
-		/* Can't allocate enough MSI-X interrupts?  Oh well.
-		 * This just means we'll go with either a single MSI
-		 * vector or fall back to legacy interrupts.
-		 */
-		netif_printk(adapter, hw, KERN_DEBUG, adapter->netdev,
-			     "Unable to allocate MSI-X interrupts\n");
-		adapter->flags &= ~IXGBE_FLAG_MSIX_ENABLED;
-		kfree(adapter->msix_entries);
-		adapter->msix_entries = NULL;
-	} else {
-		adapter->flags |= IXGBE_FLAG_MSIX_ENABLED; /* Woot! */
-		/*
-		 * Adjust for only the vectors we'll use, which is minimum
-		 * of max_msix_q_vectors + NON_Q_VECTORS, or the number of
-		 * vectors we were allocated.
-		 */
-		vectors -= NON_Q_VECTORS;
-		adapter->num_q_vectors = min(vectors, adapter->max_q_vectors);
-	}
+	vectors = min(vectors, err);
+	if (vectors < vector_threshold)
+		goto err_alloc_msix;
+
+	err = pci_enable_msix(adapter->pdev, adapter->msix_entries, vectors);
+	if (err)
+		goto err_alloc_msix;
+
+	adapter->flags |= IXGBE_FLAG_MSIX_ENABLED; /* Woot! */
+	/*
+	 * Adjust for only the vectors we'll use, which is minimum
+	 * of max_msix_q_vectors + NON_Q_VECTORS, or the number of
+	 * vectors we were allocated.
+	 */
+	vectors -= NON_Q_VECTORS;
+	adapter->num_q_vectors = min(vectors, adapter->max_q_vectors);
+
+	return;
+
+err_alloc_msix:
+	/* Can't allocate enough MSI-X interrupts?  Oh well.
+	 * This just means we'll go with either a single MSI
+	 * vector or fall back to legacy interrupts.
+	 */
+	netif_printk(adapter, hw, KERN_DEBUG, adapter->netdev,
+		     "Unable to allocate MSI-X interrupts\n");
+	adapter->flags &= ~IXGBE_FLAG_MSIX_ENABLED;
+	kfree(adapter->msix_entries);
+	adapter->msix_entries = NULL;
 }
 
 static void ixgbe_add_ring(struct ixgbe_ring *ring,
Signature · 1 line
1.7.7.6
#57 lore raw

RFC PATCH 39/77 ixgbevf: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +1 -1
index 59a62bb..fa0537a 100644
@@ -1773,7 +1773,7 @@ static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter,
 	}
 
 	if (vectors < vector_threshold)
-		err = -ENOMEM;
+		err = -ENOSPC;
 
 	if (err) {
 		dev_err(&adapter->pdev->dev,
Signature · 1 line
1.7.7.6
#58 lore raw

RFC PATCH 40/77 ixgbevf: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c |   18 +++++++-----------
 1 files changed, 7 insertions(+), 11 deletions(-)
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +7 -11
index fa0537a..d506a01 100644
@@ -1749,8 +1749,7 @@ void ixgbevf_reset(struct ixgbevf_adapter *adapter)
 static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter,
 					int vectors)
 {
-	int err = 0;
-	int vector_threshold;
+	int err, vector_threshold;
 
 	/* We'll want at least 2 (vector_threshold):
 	 * 1) TxQ[0] + RxQ[0] handler
@@ -1763,18 +1762,15 @@ static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter,
 	 * Right now, we simply care about how many we'll get; we'll
 	 * set them up later while requesting irq's.
 	 */
-	while (vectors >= vector_threshold) {
-		err = pci_enable_msix(adapter->pdev, adapter->msix_entries,
-				      vectors);
-		if (!err || err < 0) /* Success or a nasty failure. */
-			break;
-		else /* err == number of vectors we should try again with */
-			vectors = err;
-	}
+	err = pci_msix_table_size(adapter->pdev);
+	if (err < 0)
+		return err;
 
+	vectors = min(vectors, err);
 	if (vectors < vector_threshold)
-		err = -ENOSPC;
+		return -ENOSPC;
 
+	err = pci_enable_msix(adapter->pdev, adapter->msix_entries, vectors);
 	if (err) {
 		dev_err(&adapter->pdev->dev,
 			"Unable to allocate MSI-X interrupts\n");
Signature · 1 line
1.7.7.6
#60 lore raw

RFC PATCH 41/77 lpfc: Do not call pci_disable_msix() if pci_enable_msix() failed

In reply to Alexander Gordeev
---
 drivers/scsi/lpfc/lpfc_init.c |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
drivers/scsi/lpfc/lpfc_init.c +6 -3
index 647f5bf..0803b84 100644
@@ -8080,7 +8080,7 @@ lpfc_sli_enable_msix(struct lpfc_hba *phba)
 	if (rc) {
 		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
 				"0420 PCI enable MSI-X failed (%d)\n", rc);
-		goto msi_fail_out;
+		goto vec_fail_out;
 	}
 	for (i = 0; i < LPFC_MSIX_VECTORS; i++)
 		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
@@ -8158,6 +8158,8 @@ irq_fail_out:
 msi_fail_out:
 	/* Unconfigure MSI-X capability structure */
 	pci_disable_msix(phba->pcidev);
+
+vec_fail_out:
 	return rc;
 }
 
@@ -8646,7 +8648,7 @@ enable_msix_vectors:
 	} else if (rc) {
 		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
 				"0484 PCI enable MSI-X failed (%d)\n", rc);
-		goto msi_fail_out;
+		goto vec_fail_out;
 	}
 
 	/* Log MSI-X vector assignment */
@@ -8698,9 +8700,10 @@ cfg_fail_out:
 			 &phba->sli4_hba.fcp_eq_hdl[index]);
 	}
 
-msi_fail_out:
 	/* Unconfigure MSI-X capability structure */
 	pci_disable_msix(phba->pcidev);
+
+vec_fail_out:
 	return rc;
 }
 
Signature · 1 line
1.7.7.6
#61 lore raw

RFC PATCH 42/77 lpfc: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

Note, in case just one MSI-X vector was available the error message "0484 PCI enable MSI-X failed 1" is preserved to not break tools which might depend on it.

Also, not sure why in case of multiple MSI-Xs mode failed the driver skips the single MSI-X mode and falls back to single MSI mode.

---
 drivers/scsi/lpfc/lpfc_init.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)
drivers/scsi/lpfc/lpfc_init.c +11 -7
index 0803b84..d83a1a3 100644
@@ -8639,13 +8639,17 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba)
 
 	/* Configure MSI-X capability structure */
 	vectors = phba->cfg_fcp_io_channel;
-enable_msix_vectors:
-	rc = pci_enable_msix(phba->pcidev, phba->sli4_hba.msix_entries,
-			     vectors);
-	if (rc > 1) {
-		vectors = rc;
-		goto enable_msix_vectors;
-	} else if (rc) {
+
+	rc = pci_msix_table_size(phba->pcidev);
+	if (rc < 0)
+		goto msg_fail_out;
+
+	vectors = min(vectors, rc);
+	if (vectors > 1)
+		rc = pci_enable_msix(phba->pcidev, phba->sli4_hba.msix_entries,
+				     vectors);
+	if (rc) {
+msg_fail_out:
 		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
 				"0484 PCI enable MSI-X failed (%d)\n", rc);
 		goto vec_fail_out;
Signature · 1 line
1.7.7.6
#62 lore raw

RFC PATCH 43/77 lpfc: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/scsi/lpfc/lpfc_init.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
drivers/scsi/lpfc/lpfc_init.c +7 -3
index d83a1a3..0ec8008 100644
@@ -8645,9 +8645,13 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba)
 		goto msg_fail_out;
 
 	vectors = min(vectors, rc);
-	if (vectors > 1)
-		rc = pci_enable_msix(phba->pcidev, phba->sli4_hba.msix_entries,
-				     vectors);
+	if (vectors < 2) {
+		rc = -ENOSPC;
+		goto msg_fail_out;
+	}
+
+	rc = pci_enable_msix(phba->pcidev, phba->sli4_hba.msix_entries,
+			     vectors);
 	if (rc) {
 msg_fail_out:
 		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
Signature · 1 line
1.7.7.6
#63 lore raw

RFC PATCH 44/77 lpfc: Make MSI-X initialization routine more readable

In reply to Alexander Gordeev
---
 drivers/scsi/lpfc/lpfc_init.c |   23 +++++++++++------------
 1 files changed, 11 insertions(+), 12 deletions(-)
drivers/scsi/lpfc/lpfc_init.c +11 -12
index 0ec8008..0cfaf20 100644
@@ -8633,10 +8633,6 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba)
 {
 	int vectors, rc, index;
 
-	/* Set up MSI-X multi-message vectors */
-	for (index = 0; index < phba->cfg_fcp_io_channel; index++)
-		phba->sli4_hba.msix_entries[index].entry = index;
-
 	/* Configure MSI-X capability structure */
 	vectors = phba->cfg_fcp_io_channel;
 
@@ -8650,14 +8646,14 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba)
 		goto msg_fail_out;
 	}
 
+	/* Set up MSI-X multi-message vectors */
+	for (index = 0; index < vectors; index++)
+		phba->sli4_hba.msix_entries[index].entry = index;
+
 	rc = pci_enable_msix(phba->pcidev, phba->sli4_hba.msix_entries,
 			     vectors);
-	if (rc) {
-msg_fail_out:
-		lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
-				"0484 PCI enable MSI-X failed (%d)\n", rc);
-		goto vec_fail_out;
-	}
+	if (rc)
+		goto msg_fail_out;
 
 	/* Log MSI-X vector assignment */
 	for (index = 0; index < vectors; index++)
@@ -8697,7 +8693,7 @@ msg_fail_out:
 	}
 
 	lpfc_sli4_set_affinity(phba, vectors);
-	return rc;
+	return 0;
 
 cfg_fail_out:
 	/* free the irq already requested */
@@ -8710,8 +8706,11 @@ cfg_fail_out:
 
 	/* Unconfigure MSI-X capability structure */
 	pci_disable_msix(phba->pcidev);
+	return rc;
 
-vec_fail_out:
+msg_fail_out:
+	lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
+			"0484 PCI enable MSI-X failed (%d)\n", rc);
 	return rc;
 }
 
Signature · 1 line
1.7.7.6
#64 lore raw

RFC PATCH 45/77 megaraid: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/megaraid/megaraid_sas_base.c |   20 +++++++++-----------
 1 files changed, 9 insertions(+), 11 deletions(-)
drivers/scsi/megaraid/megaraid_sas_base.c +9 -11
index 3020921..b5973e4 100644
@@ -3727,18 +3727,16 @@ static int megasas_init_fw(struct megasas_instance *instance)
 					     (unsigned int)num_online_cpus());
 		for (i = 0; i < instance->msix_vectors; i++)
 			instance->msixentry[i].entry = i;
-		i = pci_enable_msix(instance->pdev, instance->msixentry,
-				    instance->msix_vectors);
-		if (i >= 0) {
-			if (i) {
-				if (!pci_enable_msix(instance->pdev,
-						     instance->msixentry, i))
-					instance->msix_vectors = i;
-				else
-					instance->msix_vectors = 0;
-			}
-		} else
+		i = pci_msix_table_size(instance->pdev);
+		if (i < 0) {
 			instance->msix_vectors = 0;
+		} else {
+			if (!pci_enable_msix(instance->pdev,
+					     instance->msixentry, i))
+				instance->msix_vectors = i;
+			else
+				instance->msix_vectors = 0;
+		}
 
 		dev_info(&instance->pdev->dev, "[scsi%d]: FW supports"
 			"<%d> MSIX vector,Online CPUs: <%d>,"
Signature · 1 line
1.7.7.6
#65 lore raw

RFC PATCH 46/77 mlx4: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/mellanox/mlx4/main.c |   17 ++++++++---------
 1 files changed, 8 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx4/main.c +8 -9
index 60c9f4f..377a5ea 100644
@@ -1852,8 +1852,16 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 	int i;
 
 	if (msi_x) {
+		err = pci_msix_table_size(dev->pdev);
+		if (err < 0)
+			goto no_msi;
+
+		/* Try if at least 2 vectors are available */
 		nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
 			     nreq);
+		nreq = min_t(int, nreq, err);
+		if (nreq < 2)
+			goto no_msi;
 
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
@@ -1862,17 +1870,8 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 		for (i = 0; i < nreq; ++i)
 			entries[i].entry = i;
 
-	retry:
 		err = pci_enable_msix(dev->pdev, entries, nreq);
 		if (err) {
-			/* Try again if at least 2 vectors are available */
-			if (err > 1) {
-				mlx4_info(dev, "Requested %d vectors, "
-					  "but only %d MSI-X vectors available, "
-					  "trying again\n", nreq, err);
-				nreq = err;
-				goto retry;
-			}
 			kfree(entries);
 			goto no_msi;
 		}
Signature · 1 line
1.7.7.6
#69 lore raw

RFC PATCH 47/77 mlx5: Fix memory leak in case not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
index b47739b..3573ba4 100644
@@ -142,6 +142,7 @@ retry:
 	}
 
 	mlx5_core_dbg(dev, "received %d MSI vectors out of %d requested\n", err, nvec);
+	kfree(table->msix_arr);
 
 	return 0;
 }
Signature · 1 line
1.7.7.6
#70 lore raw

RFC PATCH 48/77 mlx5: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -2
index 3573ba4..5e5c9a3 100644
@@ -122,7 +122,7 @@ static int mlx5_enable_msix(struct mlx5_core_dev *dev)
 	nvec = dev->caps.num_ports * num_online_cpus() + MLX5_EQ_VEC_COMP_BASE;
 	nvec = min_t(int, nvec, num_eqs);
 	if (nvec <= MLX5_EQ_VEC_COMP_BASE)
-		return -ENOMEM;
+		return -ENOSPC;
 
 	table->msix_arr = kzalloc(nvec * sizeof(*table->msix_arr), GFP_KERNEL);
 	if (!table->msix_arr)
@@ -144,7 +144,7 @@ retry:
 	mlx5_core_dbg(dev, "received %d MSI vectors out of %d requested\n", err, nvec);
 	kfree(table->msix_arr);
 
-	return 0;
+	return -ENOSPC;
 }
 
 static void mlx5_disable_msix(struct mlx5_core_dev *dev)
Signature · 1 line
1.7.7.6
#71 lore raw

RFC PATCH 49/77 mlx5: Fix minimum number of MSI-Xs

In reply to Alexander Gordeev

The minimum number of MSI-Xs is (MLX5_EQ_VEC_COMP_BASE + 1) in one check and 2 in another check. Make the checks consistent and assume the minimum number is (MLX5_EQ_VEC_COMP_BASE + 1).

---
 drivers/net/ethernet/mellanox/mlx5/core/main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -1
index 5e5c9a3..adf0e5d 100644
@@ -136,7 +136,7 @@ retry:
 	err = pci_enable_msix(dev->pdev, table->msix_arr, nvec);
 	if (err <= 0) {
 		return err;
-	} else if (err > 2) {
+	} else if (err > MLX5_EQ_VEC_COMP_BASE) {
 		nvec = err;
 		goto retry;
 	}
Signature · 1 line
1.7.7.6
#72 lore raw

RFC PATCH 50/77 mlx5: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/mellanox/mlx5/core/main.c |   18 +++++++++---------
 1 files changed, 9 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +9 -9
index adf0e5d..5c21e50 100644
@@ -119,8 +119,13 @@ static int mlx5_enable_msix(struct mlx5_core_dev *dev)
 	int err;
 	int i;
 
+	err = pci_msix_table_size(dev->pdev);
+	if (err < 0)
+		return err;
+
 	nvec = dev->caps.num_ports * num_online_cpus() + MLX5_EQ_VEC_COMP_BASE;
 	nvec = min_t(int, nvec, num_eqs);
+	nvec = min_t(int, nvec, err);
 	if (nvec <= MLX5_EQ_VEC_COMP_BASE)
 		return -ENOSPC;
 
@@ -131,20 +136,15 @@ static int mlx5_enable_msix(struct mlx5_core_dev *dev)
 	for (i = 0; i < nvec; i++)
 		table->msix_arr[i].entry = i;
 
-retry:
-	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;
 	err = pci_enable_msix(dev->pdev, table->msix_arr, nvec);
-	if (err <= 0) {
+	if (err) {
+		kfree(table->msix_arr);
 		return err;
-	} else if (err > MLX5_EQ_VEC_COMP_BASE) {
-		nvec = err;
-		goto retry;
 	}
 
-	mlx5_core_dbg(dev, "received %d MSI vectors out of %d requested\n", err, nvec);
-	kfree(table->msix_arr);
+	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;
 
-	return -ENOSPC;
+	return 0;
 }
 
 static void mlx5_disable_msix(struct mlx5_core_dev *dev)
Signature · 1 line
1.7.7.6
#76 lore raw

RFC PATCH 51/77 mthca: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/infiniband/hw/mthca/mthca_main.c |   16 +++++++++++-----
 1 files changed, 11 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mthca/mthca_main.c +11 -5
index 87897b9..3d44ca4 100644
@@ -854,17 +854,23 @@ static int mthca_enable_msi_x(struct mthca_dev *mdev)
 	struct msix_entry entries[3];
 	int err;
 
+	err = pci_msix_table_size(mdev->pdev);
+	if (err < 0)
+		return err;
+	if (err < ARRAY_SIZE(entries)) {
+		mthca_info(mdev, "Only %d MSI-X vectors available, "
+			   "not using MSI-X\n", err);
+
+		return -ENOSPC;
+	}
+
 	entries[0].entry = 0;
 	entries[1].entry = 1;
 	entries[2].entry = 2;
 
 	err = pci_enable_msix(mdev->pdev, entries, ARRAY_SIZE(entries));
-	if (err) {
-		if (err > 0)
-			mthca_info(mdev, "Only %d MSI-X vectors available, "
-				   "not using MSI-X\n", err);
+	if (err)
 		return err;
-	}
 
 	mdev->eq_table.eq[MTHCA_EQ_COMP ].msi_x_vector = entries[0].vector;
 	mdev->eq_table.eq[MTHCA_EQ_ASYNC].msi_x_vector = entries[1].vector;
Signature · 1 line
1.7.7.6
#78 lore raw

RFC PATCH 52/77 niu: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/sun/niu.c |   20 +++++++++++---------
 1 files changed, 11 insertions(+), 9 deletions(-)
drivers/net/ethernet/sun/niu.c +11 -9
index f28460c..54c41b9 100644
@@ -9051,26 +9051,28 @@ static void niu_try_msix(struct niu *np, u8 *ldg_num_map)
 		    (np->port == 0 ? 3 : 1));
 	BUG_ON(num_irqs > (NIU_NUM_LDG / parent->num_ports));
 
-retry:
+	err = pci_msix_table_size(pdev);
+	if (err < 0)
+		goto fail;
+
+	num_irqs = min(num_irqs, err);
 	for (i = 0; i < num_irqs; i++) {
 		msi_vec[i].vector = 0;
 		msi_vec[i].entry = i;
 	}
 
 	err = pci_enable_msix(pdev, msi_vec, num_irqs);
-	if (err < 0) {
-		np->flags &= ~NIU_FLAGS_MSIX;
-		return;
-	}
-	if (err > 0) {
-		num_irqs = err;
-		goto retry;
-	}
+	if (err)
+		goto fail;
 
 	np->flags |= NIU_FLAGS_MSIX;
 	for (i = 0; i < num_irqs; i++)
 		np->ldg[i].irq = msi_vec[i].vector;
 	np->num_ldg = num_irqs;
+	return;
+
+fail:
+	np->flags &= ~NIU_FLAGS_MSIX;
 }
 
 static int niu_n2_irq_init(struct niu *np, u8 *ldg_num_map)
Signature · 1 line
1.7.7.6
#79 lore raw

RFC PATCH 53/77 ntb: Fix missed call to pci_enable_msix()

In reply to Alexander Gordeev

Current MSI-X enablement code assumes MSI-Xs were successfully allocated in case less than requested vectors were available. That assumption is wrong, since MSI-Xs should be enabled with a repeated call to pci_enable_msix(). This update fixes this.

---
 drivers/ntb/ntb_hw.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
drivers/ntb/ntb_hw.c +4 -0
index 1cb6e51..de2062c 100644
@@ -1075,6 +1075,10 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 			 "Only %d MSI-X vectors.  Limiting the number of queues to that number.\n",
 			 rc);
 		msix_entries = rc;
+
+		rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
+		if (rc)
+			goto err1;
 	}
 
 	for (i = 0; i < msix_entries; i++) {
Signature · 1 line
1.7.7.6
#81 lore raw

RFC PATCH 54/77 ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Alexander Gordeev
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/ntb/ntb_hw.c +1 -1
index de2062c..eccd5e5 100644
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {
 			rc = -EIO;
 			goto err1;
 		}
Signature · 1 line
1.7.7.6
#87 lore raw

RFC PATCH 55/77 ntb: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/ntb/ntb_hw.c |   41 +++++++++++++----------------------------
 drivers/ntb/ntb_hw.h |    2 --
 2 files changed, 13 insertions(+), 30 deletions(-)
drivers/ntb/ntb_hw.c +13 -28
index eccd5e5..7776429 100644
@@ -1032,23 +1032,26 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 	struct msix_entry *msix;
 	int msix_entries;
 	int rc, i;
-	u16 val;
 
-	if (!pdev->msix_cap) {
-		rc = -EIO;
+	rc = pci_msix_table_size(pdev);
+	if (rc < 0)
 		goto err;
-	}
 
-	rc = pci_read_config_word(pdev, pdev->msix_cap + PCI_MSIX_FLAGS, &val);
-	if (rc)
+	/*
+	 * On SNB, the link interrupt is always tied to 4th vector.  If
+	 * we can't get all 4, then we can't use MSI-X.
+	 */
+	if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {
+		rc = -ENOSPC;
 		goto err;
-
-	msix_entries = msix_table_size(val);
-	if (msix_entries > ndev->limits.msix_cnt) {
+	}
+	if (rc > ndev->limits.msix_cnt) {
 		rc = -EINVAL;
 		goto err;
 	}
 
+	msix_entries = rc;
+
 	ndev->msix_entries = kmalloc(sizeof(struct msix_entry) * msix_entries,
 				     GFP_KERNEL);
 	if (!ndev->msix_entries) {
@@ -1060,26 +1063,8 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		ndev->msix_entries[i].entry = i;
 
 	rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
-	if (rc < 0)
+	if (rc)
 		goto err1;
-	if (rc > 0) {
-		/* On SNB, the link interrupt is always tied to 4th vector.  If
-		 * we can't get all 4, then we can't use MSI-X.
-		 */
-		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {
-			rc = -EIO;
-			goto err1;
-		}
-
-		dev_warn(&pdev->dev,
-			 "Only %d MSI-X vectors.  Limiting the number of queues to that number.\n",
-			 rc);
-		msix_entries = rc;
-
-		rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
-		if (rc)
-			goto err1;
-	}
 
 	for (i = 0; i < msix_entries; i++) {
 		msix = &ndev->msix_entries[i];
drivers/ntb/ntb_hw.h +0 -2
index 0a31ced..50bd760 100644
@@ -60,8 +60,6 @@
 #define PCI_DEVICE_ID_INTEL_NTB_SS_HSX		0x2F0F
 #define PCI_DEVICE_ID_INTEL_NTB_B2B_BWD		0x0C4E
 
-#define msix_table_size(control)	((control & PCI_MSIX_FLAGS_QSIZE)+1)
-
 #ifndef readq
 static inline u64 readq(void __iomem *addr)
 {
Signature · 1 line
1.7.7.6
#89 lore raw

RFC PATCH 56/77 nvme: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/block/nvme-core.c |   48 +++++++++++++++++++++++---------------------
 1 files changed, 25 insertions(+), 23 deletions(-)
drivers/block/nvme-core.c +25 -23
index da52092..f69d7af 100644
@@ -1774,34 +1774,36 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	/* Deregister the admin queue's interrupt */
 	free_irq(dev->entry[0].vector, dev->queues[0]);
 
-	vecs = nr_io_queues;
+	result = pci_msix_table_size(pdev);
+	if (result < 0)
+		goto msi;
+
+	vecs = min(result, nr_io_queues);
 	for (i = 0; i < vecs; i++)
 		dev->entry[i].entry = i;
-	for (;;) {
-		result = pci_enable_msix(pdev, dev->entry, vecs);
-		if (result <= 0)
-			break;
-		vecs = result;
-	}
 
-	if (result < 0) {
-		vecs = nr_io_queues;
-		if (vecs > 32)
-			vecs = 32;
-		for (;;) {
-			result = pci_enable_msi_block(pdev, vecs);
-			if (result == 0) {
-				for (i = 0; i < vecs; i++)
-					dev->entry[i].vector = i + pdev->irq;
-				break;
-			} else if (result < 0) {
-				vecs = 1;
-				break;
-			}
-			vecs = result;
-		}
+	result = pci_enable_msix(pdev, dev->entry, vecs);
+	if (result == 0)
+		goto irq;
+
+ msi:
+	result = pci_get_msi_cap(pdev);
+	if (result < 0)
+		goto no_msi;
+
+	vecs = min(result, nr_io_queues);
+
+	result = pci_enable_msi_block(pdev, vecs);
+	if (result == 0) {
+		for (i = 0; i < vecs; i++)
+			dev->entry[i].vector = i + pdev->irq;
+		goto irq;
 	}
 
+ no_msi:
+	vecs = 1;
+
+ irq:
 	/*
 	 * Should investigate if there's a performance win from allocating
 	 * more queues than interrupt vectors; it might allow the submission
Signature · 1 line
1.7.7.6
#90 lore raw

RFC PATCH 57/77 pmcraid: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/pmcraid.c |   23 +++++++++++------------
 1 files changed, 11 insertions(+), 12 deletions(-)
drivers/scsi/pmcraid.c +11 -12
index 1eb7b028..5c62d22 100644
@@ -4680,24 +4680,23 @@ pmcraid_register_interrupt_handler(struct pmcraid_instance *pinstance)
 
 	if ((pmcraid_enable_msix) &&
 		(pci_find_capability(pdev, PCI_CAP_ID_MSIX))) {
-		int num_hrrq = PMCRAID_NUM_MSIX_VECTORS;
 		struct msix_entry entries[PMCRAID_NUM_MSIX_VECTORS];
+		int num_hrrq = ARRAY_SIZE(entries);
 		int i;
-		for (i = 0; i < PMCRAID_NUM_MSIX_VECTORS; i++)
-			entries[i].entry = i;
-
-		rc = pci_enable_msix(pdev, entries, num_hrrq);
-		if (rc < 0)
-			goto pmcraid_isr_legacy;
 
 		/* Check how many MSIX vectors are allocated and register
 		 * msi-x handlers for each of them giving appropriate buffer
 		 */
-		if (rc > 0) {
-			num_hrrq = rc;
-			if (pci_enable_msix(pdev, entries, num_hrrq))
-				goto pmcraid_isr_legacy;
-		}
+		rc = pci_msix_table_size(pdev);
+		if (rc < 0)
+			goto pmcraid_isr_legacy;
+
+		num_hrrq = min(num_hrrq, rc);
+		for (i = 0; i < num_hrrq; i++)
+			entries[i].entry = i;
+
+		if (pci_enable_msix(pdev, entries, num_hrrq))
+			goto pmcraid_isr_legacy;
 
 		for (i = 0; i < num_hrrq; i++) {
 			pinstance->hrrq_vector[i].hrrq_id = i;
Signature · 1 line
1.7.7.6
#91 lore raw

RFC PATCH 58/77 qib: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/infiniband/hw/qib/qib_pcie.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)
drivers/infiniband/hw/qib/qib_pcie.c +0 -4
index 3f14009..9580903 100644
@@ -218,10 +218,6 @@ static void qib_msix_setup(struct qib_devdata *dd, int pos, u32 *msixcnt,
 	if (tabsize > *msixcnt)
 		tabsize = *msixcnt;
 	ret = pci_enable_msix(dd->pcidev, msix_entry, tabsize);
-	if (ret > 0) {
-		tabsize = ret;
-		ret = pci_enable_msix(dd->pcidev, msix_entry, tabsize);
-	}
 do_intx:
 	if (ret) {
 		qib_dev_err(dd,
Signature · 1 line
1.7.7.6
#92 lore raw

RFC PATCH 59/77 qla2xxx: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/scsi/qla2xxx/qla_isr.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)
drivers/scsi/qla2xxx/qla_isr.c +11 -7
index df1b30b..6c11ab9 100644
@@ -2836,16 +2836,20 @@ qla24xx_enable_msix(struct qla_hw_data *ha, struct rsp_que *rsp)
 	for (i = 0; i < ha->msix_count; i++)
 		entries[i].entry = i;
 
-	ret = pci_enable_msix(ha->pdev, entries, ha->msix_count);
-	if (ret) {
+	ret = pci_msix_table_size(ha->pdev);
+	if (ret < 0) {
+		goto msix_failed;
+	} else {
 		if (ret < MIN_MSIX_COUNT)
 			goto msix_failed;
 
-		ql_log(ql_log_warn, vha, 0x00c6,
-		    "MSI-X: Failed to enable support "
-		    "-- %d/%d\n Retry with %d vectors.\n",
-		    ha->msix_count, ret, ret);
-		ha->msix_count = ret;
+		if (ret < ha->msix_count) {
+			ql_log(ql_log_warn, vha, 0x00c6,
+			    "MSI-X: Failed to enable support "
+			    "-- %d/%d\n Retry with %d vectors.\n",
+			    ha->msix_count, ret, ret);
+			ha->msix_count = ret;
+		}
 		ret = pci_enable_msix(ha->pdev, entries, ha->msix_count);
 		if (ret) {
 msix_failed:
Signature · 1 line
1.7.7.6
#94 lore raw

RFC PATCH 60/77 qlcnic: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c +2 -1
index c4c5023..c6018bb 100644
@@ -620,7 +620,7 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 				 num_msix);
 			if (qlcnic_83xx_check(adapter)) {
 				if (err < (QLC_83XX_MINIMUM_VECTOR - tx_vector))
-					return err;
+					return -ENOSPC;
 				err -= (max_tx_rings + 1);
 				num_msix = rounddown_pow_of_two(err);
 				num_msix += (max_tx_rings + 1);
@@ -636,6 +636,7 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 					 num_msix);
 				goto enable_msix;
 			}
+			err = -ENOSPC;
 		} else {
 			dev_info(&pdev->dev,
 				 "Unable to allocate %d MSI-X interrupt vectors\n",
Signature · 1 line
1.7.7.6
#95 lore raw

RFC PATCH 61/77 qlogic: Return -EINVAL in case MSI-X is not supported

In reply to Alexander Gordeev
---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c +1 -1
index c6018bb..ff6a78b 100644
@@ -568,7 +568,7 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 {
 	struct pci_dev *pdev = adapter->pdev;
 	int max_tx_rings, max_sds_rings, tx_vector;
-	int err = -1, i;
+	int err = -EINVAL, i;
 
 	if (adapter->flags & QLCNIC_TX_INTR_SHARED) {
 		max_tx_rings = 0;
Signature · 1 line
1.7.7.6
#96 lore raw

RFC PATCH 62/77 qlcnic: Remove redundant return operator

In reply to Alexander Gordeev
---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c +0 -1
index ff6a78b..b94e679 100644
@@ -613,7 +613,6 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 				adapter->max_sds_rings = max_sds_rings;
 			}
 			dev_info(&pdev->dev, "using msi-x interrupts\n");
-			return err;
 		} else if (err > 0) {
 			dev_info(&pdev->dev,
 				 "Unable to allocate %d MSI-X interrupt vectors\n",
Signature · 1 line
1.7.7.6
#97 lore raw

RFC PATCH 63/77 qlcnic: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c |   56 ++++++++++++---------
 1 files changed, 32 insertions(+), 24 deletions(-)
drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c +32 -24
index b94e679..a137c14 100644
@@ -590,7 +590,37 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 	adapter->flags &= ~(QLCNIC_MSI_ENABLED | QLCNIC_MSIX_ENABLED);
 
 	if (adapter->ahw->msix_supported) {
- enable_msix:
+		err = pci_msix_table_size(pdev);
+		if (err < 0)
+			goto fail;
+
+		if (err < num_msix) {
+			dev_info(&pdev->dev,
+				 "Unable to allocate %d MSI-X interrupt vectors\n",
+				 num_msix);
+			if (qlcnic_83xx_check(adapter)) {
+				if (err < (QLC_83XX_MINIMUM_VECTOR - tx_vector))
+					return -ENOSPC;
+				err -= (max_tx_rings + 1);
+				num_msix = rounddown_pow_of_two(err);
+				num_msix += (max_tx_rings + 1);
+			} else {
+				num_msix = rounddown_pow_of_two(err);
+				if (qlcnic_check_multi_tx(adapter))
+					num_msix += max_tx_rings;
+			}
+
+			if (!num_msix) {
+				err = -ENOSPC;
+				goto fail;
+			}
+
+			dev_info(&pdev->dev,
+				 "Trying to allocate %d MSI-X interrupt vectors\n",
+				 num_msix);
+			}
+		}
+
 		for (i = 0; i < num_msix; i++)
 			adapter->msix_entries[i].entry = i;
 		err = pci_enable_msix(pdev, adapter->msix_entries, num_msix);
@@ -613,30 +643,8 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 				adapter->max_sds_rings = max_sds_rings;
 			}
 			dev_info(&pdev->dev, "using msi-x interrupts\n");
-		} else if (err > 0) {
-			dev_info(&pdev->dev,
-				 "Unable to allocate %d MSI-X interrupt vectors\n",
-				 num_msix);
-			if (qlcnic_83xx_check(adapter)) {
-				if (err < (QLC_83XX_MINIMUM_VECTOR - tx_vector))
-					return -ENOSPC;
-				err -= (max_tx_rings + 1);
-				num_msix = rounddown_pow_of_two(err);
-				num_msix += (max_tx_rings + 1);
-			} else {
-				num_msix = rounddown_pow_of_two(err);
-				if (qlcnic_check_multi_tx(adapter))
-					num_msix += max_tx_rings;
-			}
-
-			if (num_msix) {
-				dev_info(&pdev->dev,
-					 "Trying to allocate %d MSI-X interrupt vectors\n",
-					 num_msix);
-				goto enable_msix;
-			}
-			err = -ENOSPC;
 		} else {
+fail:
 			dev_info(&pdev->dev,
 				 "Unable to allocate %d MSI-X interrupt vectors\n",
 				 num_msix);
Signature · 1 line
1.7.7.6
#99 lore raw

RFC PATCH 64/77 qlcnic: Make MSI-X initialization routine bit more readable

In reply to Alexander Gordeev
---
 drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c |  112 +++++++++++-----------
 1 files changed, 56 insertions(+), 56 deletions(-)
drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c +56 -56
index a137c14..8510457 100644
@@ -568,7 +568,7 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 {
 	struct pci_dev *pdev = adapter->pdev;
 	int max_tx_rings, max_sds_rings, tx_vector;
-	int err = -EINVAL, i;
+	int err, i;
 
 	if (adapter->flags & QLCNIC_TX_INTR_SHARED) {
 		max_tx_rings = 0;
@@ -589,68 +589,68 @@ int qlcnic_enable_msix(struct qlcnic_adapter *adapter, u32 num_msix)
 	adapter->max_sds_rings = 1;
 	adapter->flags &= ~(QLCNIC_MSI_ENABLED | QLCNIC_MSIX_ENABLED);
 
-	if (adapter->ahw->msix_supported) {
-		err = pci_msix_table_size(pdev);
-		if (err < 0)
+	if (!adapter->ahw->msix_supported)
+		return -EINVAL;
+
+	err = pci_msix_table_size(pdev);
+	if (err < 0)
+		goto fail;
+
+	if (err < num_msix) {
+		dev_info(&pdev->dev,
+			 "Unable to allocate %d MSI-X interrupt vectors\n",
+			 num_msix);
+		if (qlcnic_83xx_check(adapter)) {
+			if (err < (QLC_83XX_MINIMUM_VECTOR - tx_vector))
+				return -ENOSPC;
+			err -= (max_tx_rings + 1);
+			num_msix = rounddown_pow_of_two(err);
+			num_msix += (max_tx_rings + 1);
+		} else {
+			num_msix = rounddown_pow_of_two(err);
+			if (qlcnic_check_multi_tx(adapter))
+				num_msix += max_tx_rings;
+		}
+
+		if (!num_msix) {
+			err = -ENOSPC;
 			goto fail;
+		}
 
-		if (err < num_msix) {
-			dev_info(&pdev->dev,
-				 "Unable to allocate %d MSI-X interrupt vectors\n",
-				 num_msix);
-			if (qlcnic_83xx_check(adapter)) {
-				if (err < (QLC_83XX_MINIMUM_VECTOR - tx_vector))
-					return -ENOSPC;
-				err -= (max_tx_rings + 1);
-				num_msix = rounddown_pow_of_two(err);
-				num_msix += (max_tx_rings + 1);
-			} else {
-				num_msix = rounddown_pow_of_two(err);
-				if (qlcnic_check_multi_tx(adapter))
-					num_msix += max_tx_rings;
-			}
+		dev_info(&pdev->dev,
+			 "Trying to allocate %d MSI-X interrupt vectors\n",
+			 num_msix);
+	}
 
-			if (!num_msix) {
-				err = -ENOSPC;
-				goto fail;
-			}
+	for (i = 0; i < num_msix; i++)
+		adapter->msix_entries[i].entry = i;
 
-			dev_info(&pdev->dev,
-				 "Trying to allocate %d MSI-X interrupt vectors\n",
-				 num_msix);
-			}
-		}
+	err = pci_enable_msix(pdev, adapter->msix_entries, num_msix);
+	if (err)
+		goto fail;
 
-		for (i = 0; i < num_msix; i++)
-			adapter->msix_entries[i].entry = i;
-		err = pci_enable_msix(pdev, adapter->msix_entries, num_msix);
-		if (err == 0) {
-			adapter->flags |= QLCNIC_MSIX_ENABLED;
-			if (qlcnic_83xx_check(adapter)) {
-				adapter->ahw->num_msix = num_msix;
-				/* subtract mail box and tx ring vectors */
-				adapter->max_sds_rings = num_msix -
-							 max_tx_rings - 1;
-			} else {
-				adapter->ahw->num_msix = num_msix;
-				if (qlcnic_check_multi_tx(adapter) &&
-				    !adapter->ahw->diag_test &&
-				    (adapter->max_drv_tx_rings > 1))
-					max_sds_rings = num_msix - max_tx_rings;
-				else
-					max_sds_rings = num_msix;
-
-				adapter->max_sds_rings = max_sds_rings;
-			}
-			dev_info(&pdev->dev, "using msi-x interrupts\n");
-		} else {
-fail:
-			dev_info(&pdev->dev,
-				 "Unable to allocate %d MSI-X interrupt vectors\n",
-				 num_msix);
-		}
+	adapter->flags |= QLCNIC_MSIX_ENABLED;
+	if (qlcnic_83xx_check(adapter)) {
+		adapter->ahw->num_msix = num_msix;
+		/* subtract mail box and tx ring vectors */
+		adapter->max_sds_rings = num_msix - max_tx_rings - 1;
+	} else {
+		adapter->ahw->num_msix = num_msix;
+		if (qlcnic_check_multi_tx(adapter) &&
+		    !adapter->ahw->diag_test && (adapter->max_drv_tx_rings > 1))
+			max_sds_rings = num_msix - max_tx_rings;
+		else
+			max_sds_rings = num_msix;
+
+		adapter->max_sds_rings = max_sds_rings;
 	}
 
+	dev_info(&pdev->dev, "using msi-x interrupts\n");
+	return 0;
+
+fail:
+	dev_info(&pdev->dev,
+		 "Unable to allocate %d MSI-X interrupt vectors\n", num_msix);
 	return err;
 }
 
Signature · 1 line
1.7.7.6
#100 lore raw

RFC PATCH 65/77 qlge: Remove a redundant assignment

In reply to Alexander Gordeev
---
 drivers/net/ethernet/qlogic/qlge/qlge_main.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
drivers/net/ethernet/qlogic/qlge/qlge_main.c +0 -1
index 2553cf4..ac54cb0 100644
@@ -3287,7 +3287,6 @@ static void ql_enable_msix(struct ql_adapter *qdev)
 			qdev->msi_x_entry = NULL;
 			netif_warn(qdev, ifup, qdev->ndev,
 				   "MSI-X Enable failed, trying MSI.\n");
-			qdev->intr_count = 1;
 			qlge_irq_type = MSI_IRQ;
 		} else if (err == 0) {
 			set_bit(QL_MSIX_ENABLED, &qdev->flags);
Signature · 1 line
1.7.7.6
#101 lore raw

RFC PATCH 66/77 qlge: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/qlogic/qlge/qlge_main.c |   39 +++++++++++--------------
 1 files changed, 17 insertions(+), 22 deletions(-)
drivers/net/ethernet/qlogic/qlge/qlge_main.c +17 -22
index ac54cb0..d6fdd93 100644
@@ -3258,45 +3258,40 @@ static void ql_enable_msix(struct ql_adapter *qdev)
 
 	/* Get the MSIX vectors. */
 	if (qlge_irq_type == MSIX_IRQ) {
+		err = pci_msix_table_size(qdev->pdev);
+		if (err < 0)
+			goto msix_fail;
+
+		qdev->intr_count = min_t(u32, qdev->intr_count, err);
+
 		/* Try to alloc space for the msix struct,
 		 * if it fails then go to MSI/legacy.
 		 */
 		qdev->msi_x_entry = kcalloc(qdev->intr_count,
 					    sizeof(struct msix_entry),
 					    GFP_KERNEL);
-		if (!qdev->msi_x_entry) {
-			qlge_irq_type = MSI_IRQ;
-			goto msi;
-		}
+		if (!qdev->msi_x_entry)
+			goto msix_fail;
 
 		for (i = 0; i < qdev->intr_count; i++)
 			qdev->msi_x_entry[i].entry = i;
 
-		/* Loop to get our vectors.  We start with
-		 * what we want and settle for what we get.
-		 */
-		do {
-			err = pci_enable_msix(qdev->pdev,
-				qdev->msi_x_entry, qdev->intr_count);
-			if (err > 0)
-				qdev->intr_count = err;
-		} while (err > 0);
-
-		if (err < 0) {
-			kfree(qdev->msi_x_entry);
-			qdev->msi_x_entry = NULL;
-			netif_warn(qdev, ifup, qdev->ndev,
-				   "MSI-X Enable failed, trying MSI.\n");
-			qlge_irq_type = MSI_IRQ;
-		} else if (err == 0) {
+		if (!pci_enable_msix(qdev->pdev,
+				     qdev->msi_x_entry, qdev->intr_count)) {
 			set_bit(QL_MSIX_ENABLED, &qdev->flags);
 			netif_info(qdev, ifup, qdev->ndev,
 				   "MSI-X Enabled, got %d vectors.\n",
 				   qdev->intr_count);
 			return;
 		}
+
+		kfree(qdev->msi_x_entry);
+		qdev->msi_x_entry = NULL;
+msix_fail:
+		netif_warn(qdev, ifup, qdev->ndev,
+			   "MSI-X Enable failed, trying MSI.\n");
+		qlge_irq_type = MSI_IRQ;
 	}
-msi:
 	qdev->intr_count = 1;
 	if (qlge_irq_type == MSI_IRQ) {
 		if (!pci_enable_msi(qdev->pdev)) {
Signature · 1 line
1.7.7.6
#102 lore raw

RFC PATCH 67/77 rapidio: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/rapidio/devices/tsi721.c |   27 +++++++++++++++++----------
 1 files changed, 17 insertions(+), 10 deletions(-)
drivers/rapidio/devices/tsi721.c +17 -10
index ff7cbf2..5edd920 100644
@@ -734,6 +734,16 @@ static int tsi721_enable_msix(struct tsi721_device *priv)
 	int err;
 	int i;
 
+	err = pci_msix_table_size(priv->pdev);
+	if (err < 0)
+		goto err_out;
+	if (err < ARRAY_SIZE(entries)) {
+		dev_info(&priv->pdev->dev,
+			 "Only %d MSI-X vectors available, not using MSI-X\n",
+			 err);
+		return -ENOSPC;
+	}
+
 	entries[TSI721_VECT_IDB].entry = TSI721_MSIX_SR2PC_IDBQ_RCV(IDB_QUEUE);
 	entries[TSI721_VECT_PWRX].entry = TSI721_MSIX_SRIO_MAC_INT;
 
@@ -769,16 +779,8 @@ static int tsi721_enable_msix(struct tsi721_device *priv)
 #endif /* CONFIG_RAPIDIO_DMA_ENGINE */
 
 	err = pci_enable_msix(priv->pdev, entries, ARRAY_SIZE(entries));
-	if (err) {
-		if (err > 0)
-			dev_info(&priv->pdev->dev,
-				 "Only %d MSI-X vectors available, "
-				 "not using MSI-X\n", err);
-		else
-			dev_err(&priv->pdev->dev,
-				"Failed to enable MSI-X (err=%d)\n", err);
-		return err;
-	}
+	if (err)
+		goto err_out;
 
 	/*
 	 * Copy MSI-X vector information into tsi721 private structure
@@ -833,6 +835,11 @@ static int tsi721_enable_msix(struct tsi721_device *priv)
 #endif /* CONFIG_RAPIDIO_DMA_ENGINE */
 
 	return 0;
+
+err_out:
+	dev_err(&priv->pdev->dev,
+		"Failed to enable MSI-X (err=%d)\n", err);
+	return err;
 }
 #endif /* CONFIG_PCI_MSI */
 
Signature · 1 line
1.7.7.6
#103 lore raw

RFC PATCH 68/77 sfc: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/sfc/efx.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)
drivers/net/ethernet/sfc/efx.c +11 -7
index 07c9bc4..184ef9f 100644
@@ -1261,21 +1261,24 @@ static int efx_probe_interrupts(struct efx_nic *efx)
 		n_channels += extra_channels;
 		n_channels = min(n_channels, efx->max_channels);
 
-		for (i = 0; i < n_channels; i++)
-			xentries[i].entry = i;
-		rc = pci_enable_msix(efx->pci_dev, xentries, n_channels);
-		if (rc > 0) {
+		rc = pci_msix_table_size(efx->pci_dev);
+		if (rc < 0)
+			goto msi;
+
+		if (rc < n_channels) {
 			netif_err(efx, drv, efx->net_dev,
 				  "WARNING: Insufficient MSI-X vectors"
 				  " available (%d < %u).\n", rc, n_channels);
 			netif_err(efx, drv, efx->net_dev,
 				  "WARNING: Performance may be reduced.\n");
-			EFX_BUG_ON_PARANOID(rc >= n_channels);
 			n_channels = rc;
-			rc = pci_enable_msix(efx->pci_dev, xentries,
-					     n_channels);
 		}
 
+		EFX_BUG_ON_PARANOID(n_channels > ARRAY_SIZE(xentries));
+		for (i = 0; i < n_channels; i++)
+			xentries[i].entry = i;
+
+		rc = pci_enable_msix(efx->pci_dev, xentries, n_channels);
 		if (rc == 0) {
 			efx->n_channels = n_channels;
 			if (n_channels > extra_channels)
@@ -1293,6 +1296,7 @@ static int efx_probe_interrupts(struct efx_nic *efx)
 				efx_get_channel(efx, i)->irq =
 					xentries[i].vector;
 		} else {
+msi:
 			/* Fall back to single channel MSI */
 			efx->interrupt_mode = EFX_INT_MODE_MSI;
 			netif_err(efx, drv, efx->net_dev,
Signature · 1 line
1.7.7.6
#104 lore raw

RFC PATCH 69/77 tg3: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/broadcom/tg3.c |   24 +++++++++++++-----------
 1 files changed, 13 insertions(+), 11 deletions(-)
drivers/net/ethernet/broadcom/tg3.c +13 -11
index 12d961c..2e842ef3 100644
@@ -11241,6 +11241,10 @@ static bool tg3_enable_msix(struct tg3 *tp)
 	int i, rc;
 	struct msix_entry msix_ent[TG3_IRQ_MAX_VECS];
 
+	rc = pci_msix_table_size(tp->pdev);
+	if (rc < 0)
+		return false;
+
 	tp->txq_cnt = tp->txq_req;
 	tp->rxq_cnt = tp->rxq_req;
 	if (!tp->rxq_cnt)
@@ -11256,6 +11260,14 @@ static bool tg3_enable_msix(struct tg3 *tp)
 		tp->txq_cnt = 1;
 
 	tp->irq_cnt = tg3_irq_count(tp);
+	if (tp->irq_cnt > rc) {
+		netdev_notice(tp->dev, "Requested %d MSI-Xs, available %d\n",
+			      tp->irq_cnt, rc);
+		tp->irq_cnt = rc;
+		tp->rxq_cnt = max(rc - 1, 1);
+		if (tp->txq_cnt)
+			tp->txq_cnt = min(tp->rxq_cnt, tp->txq_max);
+	}
 
 	for (i = 0; i < tp->irq_max; i++) {
 		msix_ent[i].entry  = i;
@@ -11263,18 +11275,8 @@ static bool tg3_enable_msix(struct tg3 *tp)
 	}
 
 	rc = pci_enable_msix(tp->pdev, msix_ent, tp->irq_cnt);
-	if (rc < 0) {
+	if (rc)
 		return false;
-	} else if (rc != 0) {
-		if (pci_enable_msix(tp->pdev, msix_ent, rc))
-			return false;
-		netdev_notice(tp->dev, "Requested %d MSI-X vectors, received %d\n",
-			      tp->irq_cnt, rc);
-		tp->irq_cnt = rc;
-		tp->rxq_cnt = max(rc - 1, 1);
-		if (tp->txq_cnt)
-			tp->txq_cnt = min(tp->rxq_cnt, tp->txq_max);
-	}
 
 	for (i = 0; i < tp->irq_max; i++)
 		tp->napi[i].irq_vec = msix_ent[i].vector;
Signature · 1 line
1.7.7.6
#105 lore raw

RFC PATCH 70/77 vmci: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/misc/vmw_vmci/vmci_guest.c |   22 +++++++++++++++-------
 1 files changed, 15 insertions(+), 7 deletions(-)
drivers/misc/vmw_vmci/vmci_guest.c +15 -7
index b3a2b76..af5caf8 100644
@@ -377,19 +377,27 @@ static int vmci_enable_msix(struct pci_dev *pdev,
 {
 	int i;
 	int result;
+	int nvec;
 
-	for (i = 0; i < VMCI_MAX_INTRS; ++i) {
+	result = pci_msix_table_size(pdev);
+	if (result < 0)
+		return result;
+
+	nvec = min(result, VMCI_MAX_INTRS);
+	if (nvec < VMCI_MAX_INTRS)
+		nvec = 1;
+
+	for (i = 0; i < nvec; ++i) {
 		vmci_dev->msix_entries[i].entry = i;
 		vmci_dev->msix_entries[i].vector = i;
 	}
 
-	result = pci_enable_msix(pdev, vmci_dev->msix_entries, VMCI_MAX_INTRS);
-	if (result == 0)
-		vmci_dev->exclusive_vectors = true;
-	else if (result > 0)
-		result = pci_enable_msix(pdev, vmci_dev->msix_entries, 1);
+	result = pci_enable_msix(pdev, vmci_dev->msix_entries, nvec);
+	if (result)
+		return result;
 
-	return result;
+	vmci_dev->exclusive_vectors = true;
+	return 0;
 }
 
 /*
Signature · 1 line
1.7.7.6
#106 lore raw

RFC PATCH 71/77 vmxnet3: Return -EINVAL if number of requested MSI-Xs is not enough

In reply to Alexander Gordeev
---
 drivers/net/vmxnet3/vmxnet3_drv.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +1 -1
index 7e2788c..5b8ea71 100644
@@ -2738,7 +2738,7 @@ static int
 vmxnet3_acquire_msix_vectors(struct vmxnet3_adapter *adapter,
 			     int vectors)
 {
-	int err = 0, vector_threshold;
+	int err = -EINVAL, vector_threshold;
 	vector_threshold = VMXNET3_LINUX_MIN_MSIX_VECT;
 
 	while (vectors >= vector_threshold) {
Signature · 1 line
1.7.7.6
#107 lore raw

RFC PATCH 72/77 vmxnet3: Fixup a weird loop exit

In reply to Alexander Gordeev
---
 drivers/net/vmxnet3/vmxnet3_drv.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +1 -1
index 5b8ea71..3518173 100644
@@ -2750,7 +2750,7 @@ vmxnet3_acquire_msix_vectors(struct vmxnet3_adapter *adapter,
 		} else if (err < 0) {
 			dev_err(&adapter->netdev->dev,
 				   "Failed to enable MSI-X, error: %d\n", err);
-			vectors = 0;
+			return err;
 		} else if (err < vector_threshold) {
 			break;
 		} else {
Signature · 1 line
1.7.7.6
#108 lore raw

RFC PATCH 73/77 vmxnet3: Return -ENOSPC when not enough MSI-X vectors available

In reply to Alexander Gordeev
---
 drivers/net/vmxnet3/vmxnet3_drv.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +4 -4
index 3518173..3df7f32 100644
@@ -2752,7 +2752,10 @@ vmxnet3_acquire_msix_vectors(struct vmxnet3_adapter *adapter,
 				   "Failed to enable MSI-X, error: %d\n", err);
 			return err;
 		} else if (err < vector_threshold) {
-			break;
+			dev_info(&adapter->pdev->dev,
+				 "Number of MSI-Xs which can be allocated "
+				 "is lower than min threshold required.\n");
+			return -ENOSPC;
 		} else {
 			/* If fails to enable required number of MSI-x vectors
 			 * try enabling minimum number of vectors required.
@@ -2764,9 +2767,6 @@ vmxnet3_acquire_msix_vectors(struct vmxnet3_adapter *adapter,
 		}
 	}
 
-	dev_info(&adapter->pdev->dev,
-		 "Number of MSI-X interrupts which can be allocated "
-		 "is lower than min threshold required.\n");
 	return err;
 }
 
Signature · 1 line
1.7.7.6
#109 lore raw

RFC PATCH 74/77 vmxnet3: Limit number of rx queues to 1 if per-queue MSI-Xs failed

In reply to Alexander Gordeev
---
 drivers/net/vmxnet3/vmxnet3_drv.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +8 -8
index 3df7f32..00dc0d0 100644
@@ -2814,12 +2814,14 @@ vmxnet3_alloc_intr_resources(struct vmxnet3_adapter *adapter)
 
 		err = vmxnet3_acquire_msix_vectors(adapter,
 						   adapter->intr.num_intrs);
-		/* If we cannot allocate one MSIx vector per queue
-		 * then limit the number of rx queues to 1
-		 */
-		if (err == VMXNET3_LINUX_MIN_MSIX_VECT) {
-			if (adapter->share_intr != VMXNET3_INTR_BUDDYSHARE
-			    || adapter->num_rx_queues != 1) {
+		if (!err) {
+			/* If we cannot allocate one MSIx vector per queue
+			 * then limit the number of rx queues to 1
+			 */
+			if ((adapter->intr.num_intrs ==
+			     VMXNET3_LINUX_MIN_MSIX_VECT) &&
+			    ((adapter->share_intr != VMXNET3_INTR_BUDDYSHARE) ||
+			     (adapter->num_rx_queues != 1))) {
 				adapter->share_intr = VMXNET3_INTR_TXSHARE;
 				netdev_err(adapter->netdev,
 					   "Number of rx queues : 1\n");
@@ -2829,8 +2831,6 @@ vmxnet3_alloc_intr_resources(struct vmxnet3_adapter *adapter)
 			}
 			return;
 		}
-		if (!err)
-			return;
 
 		/* If we cannot allocate MSIx vectors use only one rx queue */
 		dev_info(&adapter->pdev->dev,
Signature · 1 line
1.7.7.6
#110 lore raw

RFC PATCH 75/77 vmxnet3: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/vmxnet3/vmxnet3_drv.c |   68 ++++++++++++++++++-------------------
 1 files changed, 33 insertions(+), 35 deletions(-)
drivers/net/vmxnet3/vmxnet3_drv.c +33 -35
index 00dc0d0..8d3321b 100644
@@ -2724,49 +2724,47 @@ vmxnet3_read_mac_addr(struct vmxnet3_adapter *adapter, u8 *mac)
 
 #ifdef CONFIG_PCI_MSI
 
-/*
- * Enable MSIx vectors.
- * Returns :
- *	0 on successful enabling of required vectors,
- *	VMXNET3_LINUX_MIN_MSIX_VECT when only minimum number of vectors required
- *	 could be enabled.
- *	number of vectors which can be enabled otherwise (this number is smaller
- *	 than VMXNET3_LINUX_MIN_MSIX_VECT)
- */
-
 static int
 vmxnet3_acquire_msix_vectors(struct vmxnet3_adapter *adapter,
 			     int vectors)
 {
-	int err = -EINVAL, vector_threshold;
+	int err, vector_threshold;
+
 	vector_threshold = VMXNET3_LINUX_MIN_MSIX_VECT;
+	if (vectors < vector_threshold)
+		return -EINVAL;
 
-	while (vectors >= vector_threshold) {
-		err = pci_enable_msix(adapter->pdev, adapter->intr.msix_entries,
-				      vectors);
-		if (!err) {
-			adapter->intr.num_intrs = vectors;
-			return 0;
-		} else if (err < 0) {
-			dev_err(&adapter->netdev->dev,
-				   "Failed to enable MSI-X, error: %d\n", err);
-			return err;
-		} else if (err < vector_threshold) {
-			dev_info(&adapter->pdev->dev,
-				 "Number of MSI-Xs which can be allocated "
-				 "is lower than min threshold required.\n");
-			return -ENOSPC;
-		} else {
-			/* If fails to enable required number of MSI-x vectors
-			 * try enabling minimum number of vectors required.
-			 */
-			dev_err(&adapter->netdev->dev,
-				"Failed to enable %d MSI-X, trying %d instead\n",
-				    vectors, vector_threshold);
-			vectors = vector_threshold;
-		}
+	err = pci_msix_table_size(adapter->pdev);
+	if (err < 0)
+		goto err_msix;
+	if (err < vector_threshold) {
+		dev_info(&adapter->pdev->dev,
+			 "Number of MSI-X interrupts which can be allocated "
+			 "is lower than min threshold required.\n");
+		return -ENOSPC;
+	}
+	if (err < vectors) {
+		/*
+		 * If fails to enable required number of MSI-x vectors
+		 * try enabling minimum number of vectors required.
+		 */
+		dev_err(&adapter->netdev->dev,
+			"Failed to enable %d MSI-X, trying %d instead\n",
+			vectors, vector_threshold);
+		vectors = vector_threshold;
 	}
 
+	err = pci_enable_msix(adapter->pdev, adapter->intr.msix_entries,
+			      vectors);
+	if (err)
+		goto err_msix;
+
+	adapter->intr.num_intrs = vectors;
+	return 0;
+
+err_msix:
+	dev_err(&adapter->netdev->dev,
+		"Failed to enable MSI-X, error: %d\n", err);
 	return err;
 }
 
Signature · 1 line
1.7.7.6
#111 lore raw

RFC PATCH 76/77 vxge: Sanitize MSI-X allocation routine error codes

In reply to Alexander Gordeev
---
 drivers/net/ethernet/neterion/vxge/vxge-main.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)
drivers/net/ethernet/neterion/vxge/vxge-main.c +2 -4
index 5a20eaf..b81ff8b 100644
@@ -2352,7 +2352,7 @@ start:
 			"%s: MSI-X enable failed for %d vectors, ret: %d",
 			VXGE_DRIVER_NAME, vdev->intr_cnt, ret);
 		if ((max_config_vpath != VXGE_USE_DEFAULT) || (ret < 3)) {
-			ret = -ENODEV;
+			ret = -ENOSPC;
 			goto enable_msix_failed;
 		}
 
@@ -2365,10 +2365,8 @@ start:
 		vxge_close_vpaths(vdev, temp);
 		vdev->no_of_vpath = temp;
 		goto start;
-	} else if (ret < 0) {
-		ret = -ENODEV;
+	} else if (ret < 0)
 		goto enable_msix_failed;
-	}
 	return 0;
 
 enable_msix_failed:
Signature · 1 line
1.7.7.6
#112 lore raw

RFC PATCH 77/77 vxge: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/neterion/vxge/vxge-main.c |   36 ++++++++++-------------
 1 files changed, 16 insertions(+), 20 deletions(-)
drivers/net/ethernet/neterion/vxge/vxge-main.c +16 -20
index b81ff8b..b4d40dd 100644
@@ -2297,7 +2297,21 @@ static int vxge_alloc_msix(struct vxgedev *vdev)
 	int msix_intr_vect = 0, temp;
 	vdev->intr_cnt = 0;
 
-start:
+	ret = pci_msix_table_size(vdev->pdev);
+	if (ret < 0)
+		goto alloc_entries_failed;
+
+	if (ret < (vdev->no_of_vpath * 2 + 1)) {
+		if ((max_config_vpath != VXGE_USE_DEFAULT) || (ret < 3)) {
+			ret = -ENOSPC;
+			goto alloc_entries_failed;
+		}
+		/* Try with less no of vector by reducing no of vpaths count */
+		temp = (ret - 1)/2;
+		vxge_close_vpaths(vdev, temp);
+		vdev->no_of_vpath = temp;
+	}
+
 	/* Tx/Rx MSIX Vectors count */
 	vdev->intr_cnt = vdev->no_of_vpath * 2;
 
@@ -2347,25 +2361,7 @@ start:
 	vdev->vxge_entries[j].in_use = 0;
 
 	ret = pci_enable_msix(vdev->pdev, vdev->entries, vdev->intr_cnt);
-	if (ret > 0) {
-		vxge_debug_init(VXGE_ERR,
-			"%s: MSI-X enable failed for %d vectors, ret: %d",
-			VXGE_DRIVER_NAME, vdev->intr_cnt, ret);
-		if ((max_config_vpath != VXGE_USE_DEFAULT) || (ret < 3)) {
-			ret = -ENOSPC;
-			goto enable_msix_failed;
-		}
-
-		kfree(vdev->entries);
-		kfree(vdev->vxge_entries);
-		vdev->entries = NULL;
-		vdev->vxge_entries = NULL;
-		/* Try with less no of vector by reducing no of vpaths count */
-		temp = (ret - 1)/2;
-		vxge_close_vpaths(vdev, temp);
-		vdev->no_of_vpath = temp;
-		goto start;
-	} else if (ret < 0)
+	if (ret)
 		goto enable_msix_failed;
 	return 0;
 
Signature · 1 line
1.7.7.6
#48 lore raw

Re: [PATCH RFC 34/77] ioat: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

[ dropping all cc's except lkml ]

Quoted 4 lines · Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

What happens if this patch is not applied. This changelog does not have enough information for me to judge the patch. Unless I missed it I do not see a patch 00/77 explaining the background of the change, and a quick search did not turn up a discussion of why drivers now need to call pci_msix_table_size() prior to pci_enable_msix.

Quoted 2 lines

---

Not sure why everybody needed to be cc'd on this one patch. Please put targeted cc's in the patches themselves. If anything only patch 00/77 should go to the union of all addresses.

--
Dan

#49 lore raw

Re: [PATCH RFC 34/77] ioat: Update MSI/MSI-X interrupts enablement code

Quoted 8 lines · Dan J

[ dropping all cc's except lkml ]

Quoted 4 lines · Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

Quoted 5 lines

What happens if this patch is not applied. This changelog does not have enough information for me to judge the patch. Unless I missed it I do not see a patch 00/77 explaining the background of the change, and a quick search did not turn up a discussion of why drivers now need to call pci_msix_table_size() prior to pci_enable_msix.

I see patch 00/77 now and I still don't see the need for ioatdma to be updated. If we fail the subsequent request_irq, the driver will still fall back to trying less interrupts.

#9 lore raw

Re: [PATCH RFC 04/77] PCI/MSI/s390: Remove superfluous check of MSI type

In reply to Alexander Gordeev
Quoted 7 lines · Alexander Gordeev

arch_setup_msi_irqs() hook can only be called from the generic MSI code which ensures correct MSI type parameter.

---
 arch/s390/pci/pci.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

I have no idea why you sent the stable@ alias so many patches, all in the incorrect form for them to be ever accepted in the stable kernel releases. Please read Documentation/stable_kernel_rules.txt for how to do this properly.

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#54 lore raw

Re: [PATCH RFC 36/77] ipr: Enable MSI-X when IPR_USE_MSIX type is set, not IPR_USE_MSI

In reply to Alexander Gordeev
Quoted 11 lines · Alexander Gordeev
---
 drivers/scsi/ipr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/scsi/ipr.c +0 -0
index fb57e21..762a93e 100644

drivers/scsi/ipr.c static int ipr_probe_ioa(struct pci_dev *pdev, · line 9533

Quoted 8 lines
drivers/scsi/ipr.c +1 -1
@@ -9527,7 +9527,7 @@ static int ipr_probe_ioa(struct pci_dev *pdev,
 		ipr_number_of_msix = IPR_MAX_MSIX_VECTORS;
 	}
 
-	if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSI &&
+	if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSIX &&
 			ipr_enable_msix(ioa_cfg) == 0)
 		ioa_cfg->intr_flag = IPR_USE_MSIX;
 	else if (ioa_cfg->ipr_chip->intr_type == IPR_USE_MSI &&

Nack. ioa_cfg->ipr_chip->intr_type gets initialized from the ipr_chip table at the top of the driver which never sets intr_type to IPR_USE_MSIX, so this will break MSI-X support for ipr.

We indicate at the chip level only whether we want to force LSI or whether we want to enable MSI / MSI-X and then try enabling MSI-X and fall back to MSI if MSI-X is not available or does not work. We then set intr_flag to indicate what we are actually using on the specific adapter.

Thanks,

Brian

Signature · 3 lines
Brian King
Power Linux I/O
IBM Linux Technology Center
#59 lore raw

Re: [PATCH RFC 40/77] ixgbevf: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 26 lines · Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c |   18 +++++++-----------
 1 files changed, 7 insertions(+), 11 deletions(-)
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +1 -2
index fa0537a..d506a01 100644
@@ -1749,8 +1749,7 @@ void ixgbevf_reset(struct ixgbevf_adapter *adapter)
 static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter,
 					int vectors)
 {
-	int err = 0;
-	int vector_threshold;
+	int err, vector_threshold;
 
 	/* We'll want at least 2 (vector_threshold):
 	 * 1) TxQ[0] + RxQ[0] handler
@@ -1763,18 +1762,15 @@ static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter,
 	 * Right now, we simply care about how many we'll get; we'll
 	 * set them up later while requesting irq's.

drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c static int ixgbevf_acquire_msix_vectors(struct ixgbevf_adapter *adapter, · line 1765

Quoted 10 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +1 -8
 	 */
-	while (vectors >= vector_threshold) {
-		err = pci_enable_msix(adapter->pdev, adapter->msix_entries,
-				      vectors);
-		if (!err || err < 0) /* Success or a nasty failure. */
-			break;
-		else /* err == number of vectors we should try again with */
-			vectors = err;
-	}
+	err = pci_msix_table_size(adapter->pdev);

I would prefer to use something other than "err" here since the value isn't really an error if it's greater than 0. However, it's not really a big issue, since you immediately conver to using vectors on the next line of code.. I think its alright overall.

Regards,
Jake

Quoted 12 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +5 -1
+	if (err < 0)
+		return err;
 
+	vectors = min(vectors, err);
 	if (vectors < vector_threshold)
-		err = -ENOSPC;
+		return -ENOSPC;
 
+	err = pci_enable_msix(adapter->pdev, adapter->msix_entries, vectors);
 	if (err) {
 		dev_err(&adapter->pdev->dev,
 			"Unable to allocate MSI-X interrupts\n");
#113 lore raw

Re: [PATCH RFC 77/77] vxge: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 22 lines · Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/neterion/vxge/vxge-main.c |   36 ++++++++++-------------
 1 files changed, 16 insertions(+), 20 deletions(-)
drivers/net/ethernet/neterion/vxge/vxge-main.c +4 -1
index b81ff8b..b4d40dd 100644
@@ -2297,7 +2297,21 @@ static int vxge_alloc_msix(struct vxgedev *vdev)
 	int msix_intr_vect = 0, temp;
 	vdev->intr_cnt = 0;
 
-start:
+	ret = pci_msix_table_size(vdev->pdev);
+	if (ret < 0)
+		goto alloc_entries_failed;
+

drivers/net/ethernet/neterion/vxge/vxge-main.c static int vxge_alloc_msix(struct vxgedev *vdev) · line 2313

Quoted 10 lines
drivers/net/ethernet/neterion/vxge/vxge-main.c +10 -0
+	if (ret < (vdev->no_of_vpath * 2 + 1)) {
+		if ((max_config_vpath != VXGE_USE_DEFAULT) || (ret < 3)) {
+			ret = -ENOSPC;
+			goto alloc_entries_failed;
+		}
+		/* Try with less no of vector by reducing no of vpaths count */
+		temp = (ret - 1)/2;
+		vxge_close_vpaths(vdev, temp);
+		vdev->no_of_vpath = temp;
+	}

The original code was ugly (not my code, so I can say that). I'd like to see it a little stream lined. Something like:

        vdev->intr_cnt = pci_msix_table_size(vdev->pdev);

        if (vdev->intr_cnt % 2 == 0)
                vdev->intr_cnt--;

        if (vdev->intr_cnt < 3 || max_config_vpath != VXGE_USE_DEFAULT)
                goto alloc_entries_failed;

        if (vdev->intr_cnt != vdev->no_of_vpath * 2 + 1) {
                vxge_close_vpaths(vdev, vdev->intr_cnt / 2);
                vdev->no_of_vpath = vdev->intr_cnt / 2;
        }
Quoted 19 lines
drivers/net/ethernet/neterion/vxge/vxge-main.c +1 -11
+
 	/* Tx/Rx MSIX Vectors count */
 	vdev->intr_cnt = vdev->no_of_vpath * 2;
 
@@ -2347,25 +2361,7 @@ start:
 	vdev->vxge_entries[j].in_use = 0;
 
 	ret = pci_enable_msix(vdev->pdev, vdev->entries, vdev->intr_cnt);
-	if (ret > 0) {
-		vxge_debug_init(VXGE_ERR,
-			"%s: MSI-X enable failed for %d vectors, ret: %d",
-			VXGE_DRIVER_NAME, vdev->intr_cnt, ret);
-		if ((max_config_vpath != VXGE_USE_DEFAULT) || (ret < 3)) {
-			ret = -ENOSPC;
-			goto enable_msix_failed;
-		}
-
-		kfree(vdev->entries);
-		kfree(vdev->vxge_entries);

drivers/net/ethernet/neterion/vxge/vxge-main.c start: · line 2365

Quoted 10 lines
drivers/net/ethernet/neterion/vxge/vxge-main.c +1 -8
-		vdev->entries = NULL;
-		vdev->vxge_entries = NULL;
-		/* Try with less no of vector by reducing no of vpaths count */
-		temp = (ret - 1)/2;
-		vxge_close_vpaths(vdev, temp);
-		vdev->no_of_vpath = temp;
-		goto start;
-	} else if (ret < 0)
+	if (ret)
 		goto enable_msix_failed;

Nit, space here please.

Quoted 5 lines
drivers/net/ethernet/neterion/vxge/vxge-main.c +0 -0
 	return 0;
 
Signature · 1 line
1.7.7.6
#2 lore raw

Re: [PATCH RFC 01/77] PCI/MSI: Fix return value when populate_msi_sysfs() failed

In reply to Alexander Gordeev
Quoted 1 line · Alexander Gordeev

Since you are changing the behavior of the msix_capability_init function on populate_msi_sysfs error, a comment describing why in this commit would be nice.

Quoted 50 lines
---
 drivers/pci/msi.c |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)
drivers/pci/msi.c +5 -6
index d5f90d6..b43f391 100644
@@ -719,7 +719,7 @@ static int msix_capability_init(struct pci_dev *dev,
 
 	ret = arch_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
 	if (ret)
-		goto error;
+		goto out_avail;
 
 	/*
 	 * Some devices require MSI-X to be enabled before we can touch the
@@ -732,10 +732,8 @@ static int msix_capability_init(struct pci_dev *dev,
 	msix_program_entries(dev, entries);
 
 	ret = populate_msi_sysfs(dev);
-	if (ret) {
-		ret = 0;
-		goto error;
-	}
+	if (ret)
+		goto out_free;
 
 	/* Set MSI-X enabled bits and unmask the function */
 	pci_intx_for_msi(dev, 0);
@@ -746,7 +744,7 @@ static int msix_capability_init(struct pci_dev *dev,
 
 	return 0;
 
-error:
+out_avail:
 	if (ret < 0) {
 		/*
 		 * If we had some success, report the number of irqs
@@ -763,6 +761,7 @@ error:
 			ret = avail;
 	}
 
+out_free:
 	free_msi_irqs(dev);
 
 	return ret;
Signature · 1 line
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#82 lore raw

Re: [PATCH RFC 54/77] ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Alexander Gordeev
Quoted 10 lines · Alexander Gordeev
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/ntb/ntb_hw.c +0 -0
index de2062c..eccd5e5 100644

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1069

Quoted 5 lines
drivers/ntb/ntb_hw.c +1 -1
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Nack, this check is unnecessary.

Also, no comment in the commit on why it could be necessary.

Quoted 6 lines
drivers/ntb/ntb_hw.c +0 -0
 			rc = -EIO;
 			goto err1;
 		}
Signature · 1 line
1.7.7.6

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

#80 lore raw

Re: [PATCH RFC 53/77] ntb: Fix missed call to pci_enable_msix()

In reply to Alexander Gordeev
Quoted 4 lines · Alexander Gordeev

Current MSI-X enablement code assumes MSI-Xs were successfully allocated in case less than requested vectors were available. That assumption is wrong, since MSI-Xs should be enabled with a repeated call to pci_enable_msix(). This update fixes this.

Good catch, I'll pull it in for my next NTB release.

Thanks,
Jon

Quoted 24 lines
---
 drivers/ntb/ntb_hw.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
drivers/ntb/ntb_hw.c +4 -0
index 1cb6e51..de2062c 100644
@@ -1075,6 +1075,10 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 			 "Only %d MSI-X vectors.  Limiting the number of queues to that number.\n",
 			 rc);
 		msix_entries = rc;
+
+		rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
+		if (rc)
+			goto err1;
 	}
 
 	for (i = 0; i < msix_entries; i++) {
Signature · 1 line
1.7.7.6
#88 lore raw

Re: [PATCH RFC 55/77] ntb: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 24 lines · Alexander Gordeev

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/ntb/ntb_hw.c |   41 +++++++++++++----------------------------
 drivers/ntb/ntb_hw.h |    2 --
 2 files changed, 13 insertions(+), 30 deletions(-)
drivers/ntb/ntb_hw.c +2 -3
index eccd5e5..7776429 100644
@@ -1032,23 +1032,26 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 	struct msix_entry *msix;
 	int msix_entries;
 	int rc, i;
-	u16 val;
 
-	if (!pdev->msix_cap) {
-		rc = -EIO;
+	rc = pci_msix_table_size(pdev);
+	if (rc < 0)

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1044

Quoted 10 lines
drivers/ntb/ntb_hw.c +5 -3
 		goto err;
-	}
 
-	rc = pci_read_config_word(pdev, pdev->msix_cap + PCI_MSIX_FLAGS, &val);
-	if (rc)
+	/*
+	 * On SNB, the link interrupt is always tied to 4th vector.  If
+	 * we can't get all 4, then we can't use MSI-X.
+	 */
+	if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Please check for the HW type first, and then compare to ndev->limits.msix_cnt (which will be SNB_MSIX_CNT on SNB HW). Also, put the comment inside the if statement and remove the unecessary "()" around the comparisons. OCD on my part, but I like it that way.

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1047

Quoted 6 lines
drivers/ntb/ntb_hw.c +2 -3
+		rc = -ENOSPC;
 		goto err;
-
-	msix_entries = msix_table_size(val);
-	if (msix_entries > ndev->limits.msix_cnt) {
+	}

else if...

Quoted 43 lines
drivers/ntb/ntb_hw.c +4 -19
+	if (rc > ndev->limits.msix_cnt) {
 		rc = -EINVAL;
 		goto err;
 	}
 
+	msix_entries = rc;
+
 	ndev->msix_entries = kmalloc(sizeof(struct msix_entry) * msix_entries,
 				     GFP_KERNEL);
 	if (!ndev->msix_entries) {
@@ -1060,26 +1063,8 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		ndev->msix_entries[i].entry = i;
 
 	rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
-	if (rc < 0)
+	if (rc)
 		goto err1;
-	if (rc > 0) {
-		/* On SNB, the link interrupt is always tied to 4th vector.  If
-		 * we can't get all 4, then we can't use MSI-X.
-		 */
-		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {
-			rc = -EIO;
-			goto err1;
-		}
-
-		dev_warn(&pdev->dev,
-			 "Only %d MSI-X vectors.  Limiting the number of queues to that number.\n",
-			 rc);
-		msix_entries = rc;
-
-		rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);
-		if (rc)
-			goto err1;
-	}
 
 	for (i = 0; i < msix_entries; i++) {
 		msix = &ndev->msix_entries[i];
drivers/ntb/ntb_hw.h +0 -0
index 0a31ced..50bd760 100644

drivers/ntb/ntb_hw.h line 63

Quoted 4 lines
drivers/ntb/ntb_hw.h +0 -1
@@ -60,8 +60,6 @@
 #define PCI_DEVICE_ID_INTEL_NTB_SS_HSX		0x2F0F
 #define PCI_DEVICE_ID_INTEL_NTB_B2B_BWD		0x0C4E
 
-#define msix_table_size(control)	((control & PCI_MSIX_FLAGS_QSIZE)+1)

Good riddance! :-)

Quoted 7 lines
drivers/ntb/ntb_hw.h +0 -1
-
 #ifndef readq
 static inline u64 readq(void __iomem *addr)
 {
Signature · 1 line
1.7.7.6
#73 lore raw

Re: [PATCH RFC 50/77] mlx5: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

drivers/net/ethernet/mellanox/mlx5/core/main.c static int mlx5_enable_msix(struct mlx5_core_dev *dev) · line 130

Quoted 10 lines · Alexander Gordeev
drivers/net/ethernet/mellanox/mlx5/core/main.c +5 -0
+	err = pci_msix_table_size(dev->pdev);
+	if (err < 0)
+		return err;
+
 	nvec = dev->caps.num_ports * num_online_cpus() + MLX5_EQ_VEC_COMP_BASE;
 	nvec = min_t(int, nvec, num_eqs);
+	nvec = min_t(int, nvec, err);
 	if (nvec <= MLX5_EQ_VEC_COMP_BASE)
 		return -ENOSPC;

Making sure we don't request more vectors then the device's is capable of -- looks good.

Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +0 -2
@@ -131,20 +136,15 @@ static int mlx5_enable_msix(struct mlx5_core_dev *dev)
 	for (i = 0; i < nvec; i++)
 		table->msix_arr[i].entry = i;
 
-retry:
-	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;

drivers/net/ethernet/mellanox/mlx5/core/main.c static int mlx5_enable_msix(struct mlx5_core_dev *dev) · line 144

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -4
 	err = pci_enable_msix(dev->pdev, table->msix_arr, nvec);
-	if (err <= 0) {
+	if (err) {
+		kfree(table->msix_arr);
 		return err;
-	} else if (err > MLX5_EQ_VEC_COMP_BASE) {
-		nvec = err;
-		goto retry;
 	}
 

According to latest sources, pci_enable_msix() may still fail so why do you want to remove this code?

Quoted 16 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -3
-	mlx5_core_dbg(dev, "received %d MSI vectors out of %d requested\n", err, nvec);
-	kfree(table->msix_arr);
+	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;
 
-	return -ENOSPC;
+	return 0;
 }
 
 static void mlx5_disable_msix(struct mlx5_core_dev *dev)
Signature · 6 lines
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#66 lore raw

Re: [PATCH RFC 46/77] mlx4: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

On Wed, 2 Oct 2013 12:49:02 +0200
Alexander Gordeev <[email protected]> wrote:

NACK. This change does not do anything logically as far as I can tell. pci_enable_msix in the current upstream kernel itself calls pci_msix_table_size. The current code yields the same results as the code suggested below. (i.e., the suggested code has no effect on optimality).

BTW, pci_msix_table_size never returns a value < 0 (if msix is not enabled, it returns 0 for the table size), so the (err < 0) check here is not correct. (I also do not like using "err" here anyway for the value returned by pci_msix_table_size(). There is no error here, and it is simply confusing.

-Jack

Quoted 52 lines

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/mellanox/mlx4/main.c |   17 ++++++++---------
 1 files changed, 8 insertions(+), 9 deletions(-)
(quoted diff) +0 -0

b/drivers/net/ethernet/mellanox/mlx4/main.c index 60c9f4f..377a5ea 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c

drivers/net/ethernet/mellanox/mlx4/main.c +8 -2
@@ -1852,8 +1852,16 @@ static void mlx4_enable_msi_x(struct mlx4_dev
 *dev) int i;
 
 	if (msi_x) {
+		err = pci_msix_table_size(dev->pdev);
+		if (err < 0)
+			goto no_msi;
+
+		/* Try if at least 2 vectors are available */
 		nreq = min_t(int, dev->caps.num_eqs -
 dev->caps.reserved_eqs, nreq);
+		nreq = min_t(int, nreq, err);
+		if (nreq < 2)
+			goto no_msi;
 
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
@@ -1862,17 +1870,8 @@ static void mlx4_enable_msi_x(struct mlx4_dev
 *dev) for (i = 0; i < nreq; ++i)
 			entries[i].entry = i;
 
-	retry:
 		err = pci_enable_msix(dev->pdev, entries, nreq);
 		if (err) {
-			/* Try again if at least 2 vectors are

available */

(quoted diff) +0 -2
-			if (err > 1) {
-				mlx4_info(dev, "Requested %d

vectors, "

(quoted diff) +0 -1
-					  "but only %d MSI-X vectors

available, "

(quoted diff) +0 -1
-					  "trying again\n", nreq,

err);

drivers/net/ethernet/mellanox/mlx4/main.c +0 -3
-				nreq = err;
-				goto retry;
-			}
 			kfree(entries);
 			goto no_msi;
 		}
#67 lore raw

Re: [PATCH RFC 46/77] mlx4: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev

On Wed, 2 Oct 2013 12:49:02 +0200
Alexander Gordeev <[email protected]> wrote:

UPDATING THIS REPLY.
Your change log confused me. The change below is not from a "recent re-design", it is required due to an earlier patch in this patch set.

Quoted 1 line

From the log, I assumed that the change you are talking about is already

upstream.

I will re-review.

-Jack

NACK. This change does not do anything logically as far as I can tell. pci_enable_msix in the current upstream kernel itself calls pci_msix_table_size. The current code yields the same resultswill as the code suggested below. (i.e., the suggested code has no effect on optimality).

BTW, pci_msix_table_size never returns a value < 0 (if msix is not enabled, it returns 0 for the table size), so the (err < 0) check here is not correct. (I also do not like using "err" here anyway for the value returned by pci_msix_table_size(). There is no error here, and it is simply confusing.

-Jack

Quoted 52 lines

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
 drivers/net/ethernet/mellanox/mlx4/main.c |   17 ++++++++---------
 1 files changed, 8 insertions(+), 9 deletions(-)
(quoted diff) +0 -0

b/drivers/net/ethernet/mellanox/mlx4/main.c index 60c9f4f..377a5ea 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c

drivers/net/ethernet/mellanox/mlx4/main.c +8 -2
@@ -1852,8 +1852,16 @@ static void mlx4_enable_msi_x(struct mlx4_dev
 *dev) int i;
 
 	if (msi_x) {
+		err = pci_msix_table_size(dev->pdev);
+		if (err < 0)
+			goto no_msi;
+
+		/* Try if at least 2 vectors are available */
 		nreq = min_t(int, dev->caps.num_eqs -
 dev->caps.reserved_eqs, nreq);
+		nreq = min_t(int, nreq, err);
+		if (nreq < 2)
+			goto no_msi;
 
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
@@ -1862,17 +1870,8 @@ static void mlx4_enable_msi_x(struct mlx4_dev
 *dev) for (i = 0; i < nreq; ++i)
 			entries[i].entry = i;
 
-	retry:
 		err = pci_enable_msix(dev->pdev, entries, nreq);
 		if (err) {
-			/* Try again if at least 2 vectors are

available */

(quoted diff) +0 -2
-			if (err > 1) {
-				mlx4_info(dev, "Requested %d

vectors, "

(quoted diff) +0 -1
-					  "but only %d MSI-X vectors

available, "

(quoted diff) +0 -1
-					  "trying again\n", nreq,

err);

drivers/net/ethernet/mellanox/mlx4/main.c +0 -3
-				nreq = err;
-				goto retry;
-			}
 			kfree(entries);
 			goto no_msi;
 		}
#68 lore raw

Re: [PATCH RFC 46/77] mlx4: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 5 lines · On Wed, 2 Oct 2013 12:49:02 +0200 Alexander Gordeev <[email protected]>…

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

New review -- ACK (i.e., patch is OK), subject to acceptance of patches 05 and 07 of this patch set.

I sent my previous review (NACK) when I was not yet aware that changes proposed were due to the two earlier patches (mentioned above) in the current patch set.

The change log here should actually read something like the following:

As a result of changes to the MSI/MSI_X enabling procedures, this driver must be modified in order to preserve its current msi/msi_x enablement logic.

-Jack

#77 lore raw

Re: [PATCH RFC 51/77] mthca: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 11 lines · On Wed, 2 Oct 2013 12:49:07 +0200 Alexander Gordeev <[email protected]>…

Subject: [PATCH RFC 51/77] mthca: Update MSI/MSI-X interrupts enablement code Date: Wed, 2 Oct 2013 12:49:07 +0200
Sender: [email protected]
X-Mailer: git-send-email 1.7.7.6

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---

ACK.

-Jack

Acked-by on v1 59/77
#93 lore raw

Re: [PATCH RFC 59/77] qla2xxx: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 50 lines

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

---
drivers/scsi/qla2xxx/qla_isr.c | 18 +++++++++++-------

1 files changed, 11 insertions(+), 7 deletions(-)
drivers/scsi/qla2xxx/qla_isr.c +11 -7
index df1b30b..6c11ab9 100644
@@ -2836,16 +2836,20 @@ qla24xx_enable_msix(struct qla_hw_data *ha,
 struct rsp_que *rsp)
 	for (i = 0; i < ha->msix_count; i++)
 		entries[i].entry = i;
 
-	ret = pci_enable_msix(ha->pdev, entries, ha->msix_count);
-	if (ret) {
+	ret = pci_msix_table_size(ha->pdev);
+	if (ret < 0) {
+		goto msix_failed;
+	} else {
 		if (ret < MIN_MSIX_COUNT)
 			goto msix_failed;
 
-		ql_log(ql_log_warn, vha, 0x00c6,
-		    "MSI-X: Failed to enable support "
-		    "-- %d/%d\n Retry with %d vectors.\n",
-		    ha->msix_count, ret, ret);
-		ha->msix_count = ret;
+		if (ret < ha->msix_count) {
+			ql_log(ql_log_warn, vha, 0x00c6,
+			    "MSI-X: Failed to enable support "
+			    "-- %d/%d\n Retry with %d vectors.\n",
+			    ha->msix_count, ret, ret);
+			ha->msix_count = ret;
+		}
 		ret = pci_enable_msix(ha->pdev, entries, ha->msix_count);
 		if (ret) {

msix_failed:

Signature · 6 lines
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#74 lore raw

Re: [PATCH RFC 50/77] mlx5: Update MSI/MSI-X interrupts enablement code

In reply to Eli Cohen
Quoted 19 lines · Eli Cohen

drivers/net/ethernet/mellanox/mlx5/core/main.c static int mlx5_enable_msix(struct mlx5_core_dev *dev) · line 130

Quoted 10 lines · Alexander Gordeev
drivers/net/ethernet/mellanox/mlx5/core/main.c +5 -0
+	err = pci_msix_table_size(dev->pdev);
+	if (err < 0)
+		return err;
+
 	nvec = dev->caps.num_ports * num_online_cpus() + MLX5_EQ_VEC_COMP_BASE;
 	nvec = min_t(int, nvec, num_eqs);
+	nvec = min_t(int, nvec, err);
 	if (nvec <= MLX5_EQ_VEC_COMP_BASE)
 		return -ENOSPC;

Making sure we don't request more vectors then the device's is capable of -- looks good.

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +0 -0
@@ -131,20 +136,15 @@ static int mlx5_enable_msix(struct mlx5_core_dev *dev)
 	for (i = 0; i < nvec; i++)
 		table->msix_arr[i].entry = i;
 
Quoted 15 lines

drivers/net/ethernet/mellanox/mlx5/core/main.c static int mlx5_enable_msix(struct mlx5_core_dev *dev) · line 144

Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -6
-retry:
-	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;
 	err = pci_enable_msix(dev->pdev, table->msix_arr, nvec);
-	if (err <= 0) {
+	if (err) {
+		kfree(table->msix_arr);
 		return err;
-	} else if (err > MLX5_EQ_VEC_COMP_BASE) {
-		nvec = err;
-		goto retry;
 	}
 

According to latest sources, pci_enable_msix() may still fail so why do you want to remove this code?

pci_enable_msix() may fail, but it can not return a positive number.

We first calculate how many MSI-Xs we need, adjust to what we can get, check if that is enough and only then go for it.

Quoted 9 lines
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -3
-	mlx5_core_dbg(dev, "received %d MSI vectors out of %d requested\n", err, nvec);
-	kfree(table->msix_arr);
+	table->num_comp_vectors = nvec - MLX5_EQ_VEC_COMP_BASE;
 
-	return -ENOSPC;
+	return 0;
 }
 
 static void mlx5_disable_msix(struct mlx5_core_dev *dev)
Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#50 lore raw

Re: [PATCH RFC 34/77] ioat: Update MSI/MSI-X interrupts enablement code

In reply to Williams, Dan J
Quoted 3 lines · Dan J

I see patch 00/77 now and I still don't see the need for ioatdma to be updated. If we fail the subsequent request_irq, the driver will still fall back to trying less interrupts.

In the proposed design pci_enable_msix() will never return a positive value. Therefore if the driver is not updated it will fallback to single MSI if device->common.chancnt MSI-X vectors were not allocated while it used to fallback to a single MSI-X in the same situation. Not to mention 'if (err > 0)' dead code.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#51 lore raw

Re: [PATCH RFC 34/77] ioat: Update MSI/MSI-X interrupts enablement code

Quoted 9 lines · Alexander Gordeev
Quoted 3 lines · Dan J

I see patch 00/77 now and I still don't see the need for ioatdma to be updated. If we fail the subsequent request_irq, the driver will still fall back to trying less interrupts.

In the proposed design pci_enable_msix() will never return a positive value. Therefore if the driver is not updated it will fallback to single MSI if device->common.chancnt MSI-X vectors were not allocated while it used to fallback to a single MSI-X in the same situation.

That's fine. I just don't like how this patch makes the driver more complex. If we're making things simpler lets not expose msix internal details and just fallback to msi directly without worrying about the number of vectors. I think the msix_single_vector intermediate step should go.

...and now that I look I see a bug in ioat3_irq_reinit.

Quoted 2 lines

Not to mention 'if (err > 0)'
dead code.

Ok, then just this piece for now, but more preferable to fold it into a msix_single_vector removal patch.

- if (err < 0)
+ if (err)

                goto msi;

- if (err > 0)
- goto msix_single_vector;

--
Dan

#3 lore raw

Re: [PATCH RFC 01/77] PCI/MSI: Fix return value when populate_msi_sysfs() failed

In reply to Jon Mason
Quoted 6 lines · Jon Mason
Quoted 1 line · Alexander Gordeev

Since you are changing the behavior of the msix_capability_init function on populate_msi_sysfs error, a comment describing why in this commit would be nice.

[...]

This function was already treating that error as fatal, and freeing the MSIs. The change in behaviour is that it now returns the error code in this case, rather than 0. This is obviously correct and properly described by the one-line summary.

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#15 lore raw

Re: [PATCH RFC 06/77] PCI/MSI: Factor out pci_get_msi_cap() interface

In reply to Alexander Gordeev

On Wed, 2013-10-02 at 12:48 +0200, Alexander Gordeev wrote:
[...]

Quoted 13 lines
drivers/pci/msi.c +7 -0
@@ -812,6 +812,21 @@ static int pci_msi_check_device(struct pci_dev *dev, int nvec, int type)
 	return 0;
 }
 
+int pci_get_msi_cap(struct pci_dev *dev)
+{
+	int ret;
+	u16 msgctl;
+
+	if (!dev->msi_cap)
+		return -EINVAL;

[...]

include/linux/pci.h struct msix_entry { · line 1149

Quoted 9 lines
include/linux/pci.h +3 -0
@@ -1144,6 +1144,11 @@ struct msix_entry {
 
 
 #ifndef CONFIG_PCI_MSI
+static inline int pci_get_msi_cap(struct pci_dev *dev)
+{
+	return -1;

[...]

Shouldn't this also return -EINVAL?

Ben.

Signature · 8 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#114 lore raw
In reply to Alexander Gordeev
Quoted 25 lines · Alexander Gordeev

This series is against "next" branch in Bjorn's repo:
git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git

Currently pci_enable_msi_block() and pci_enable_msix() interfaces return a error code in case of failure, 0 in case of success and a positive value which indicates the number of MSI-X/MSI interrupts that could have been allocated. The latter value should be passed to a repeated call to the interfaces until a failure or success:

	for (i = 0; i < FOO_DRIVER_MAXIMUM_NVEC; i++)
		adapter->msix_entries[i].entry = i;

	while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
		rc = pci_enable_msix(adapter->pdev,
				     adapter->msix_entries, nvec);
		if (rc > 0)
			nvec = rc;
		else
			return rc;
	}

	return -ENOSPC;
Quoted 6 lines

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

[...]

I think this is fundamentally flawed: pci_msix_table_size() and pci_get_msi_cap() can only report the limits of the *device* (which the driver usually already knows), whereas MSI allocation can also be constrained due to *global* limits on the number of distinct IRQs.

Currently pci_enable_msix() will report a positive value if it fails due to the global limit. Your patch 7 removes that. pci_enable_msi_block() unfortunately doesn't appear to do this.

It seems to me that a more useful interface would take a minimum and maximum number of vectors from the driver. This wouldn't allow the driver to specify that it could only accept, say, any even number within a certain range, but you could still leave the current functions available for any driver that needs that.

Ben.

Signature · 8 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw

Re: [PATCH RFC 01/77] PCI/MSI: Fix return value when populate_msi_sysfs() failed

In reply to Ben Hutchings
Quoted 7 lines · Ben Hutchings
Quoted 6 lines · Jon Mason
Quoted 1 line · Alexander Gordeev

Since you are changing the behavior of the msix_capability_init function on populate_msi_sysfs error, a comment describing why in this commit would be nice.

Quoted 6 lines

[...]

This function was already treating that error as fatal, and freeing the MSIs. The change in behaviour is that it now returns the error code in this case, rather than 0. This is obviously correct and properly described by the one-line summary.

If someone dumb, like me, is looking at this commit and trying to figure out what is happening, having ANY commit message is good. "Fix the return value" doesn't tell me anything. Documenting what issue(s) would've been seen had the error case been encountered and what will now been seen would be very nice.

Quoted 8 lines

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#16 lore raw

Re: [PATCH RFC 06/77] PCI/MSI: Factor out pci_get_msi_cap() interface

In reply to Ben Hutchings
Quoted 8 lines · Ben Hutchings

include/linux/pci.h struct msix_entry { · line 1149

Quoted 4 lines · Alexander Gordeev

#ifndef CONFIG_PCI_MSI

include/linux/pci.h +3 -0
+static inline int pci_get_msi_cap(struct pci_dev *dev)
+{
+	return -1;

[...]

Shouldn't this also return -EINVAL?

Yep, all inliners here are better to return -EINVAL. Will do unless someone speaks out against.

Quoted 1 line

Ben.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
Acked-by on v1 3/77
#7 lore raw

Re: [PATCH RFC 03/77] PCI/MSI/s390: Fix single MSI only check

In reply to Alexander Gordeev
Quoted 13 lines · On Wed, 2 Oct 2013 12:48:19 +0200 Alexander Gordeev <[email protected]>…

Multiple MSIs have never been supported on s390 architecture, but the platform code fails to report single MSI only.

---
 arch/s390/pci/pci.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/s390/pci/pci.c +0 -0
index f17a834..c79c6e4 100644

arch/s390/pci/pci.c int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) · line 434

Quoted 8 lines
arch/s390/pci/pci.c +2 -0
@@ -427,6 +427,8 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 	pr_debug("%s: requesting %d MSI-X interrupts...", __func__, nvec);
 	if (type != PCI_CAP_ID_MSIX && type != PCI_CAP_ID_MSI)
 		return -EINVAL;
+	if (type == PCI_CAP_ID_MSI && nvec > 1)
+		return 1;
 	msi_vecs = min(nvec, ZPCI_MSI_VEC_MAX);
 	msi_vecs = min_t(unsigned int, msi_vecs, CONFIG_PCI_NR_MSI);
 
Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
Acked-by on v1 4/77
#10 lore raw

Re: [PATCH RFC 04/77] PCI/MSI/s390: Remove superfluous check of MSI type

In reply to Alexander Gordeev
Quoted 13 lines · On Wed, 2 Oct 2013 12:48:20 +0200 Alexander Gordeev <[email protected]>…

arch_setup_msi_irqs() hook can only be called from the generic MSI code which ensures correct MSI type parameter.

---
 arch/s390/pci/pci.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)
arch/s390/pci/pci.c +0 -0
index c79c6e4..61a3c2c 100644

arch/s390/pci/pci.c int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) · line 430

Quoted 8 lines
arch/s390/pci/pci.c +0 -2
@@ -425,8 +425,6 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 	int rc;
 
 	pr_debug("%s: requesting %d MSI-X interrupts...", __func__, nvec);
-	if (type != PCI_CAP_ID_MSIX && type != PCI_CAP_ID_MSI)
-		return -EINVAL;
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
 		return 1;
 	msi_vecs = min(nvec, ZPCI_MSI_VEC_MAX);
Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#115 lore raw
In reply to Ben Hutchings
Quoted 10 lines · Ben Hutchings
Quoted 3 lines · Alexander Gordeev

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

[...]

I think this is fundamentally flawed: pci_msix_table_size() and pci_get_msi_cap() can only report the limits of the *device* (which the driver usually already knows), whereas MSI allocation can also be constrained due to *global* limits on the number of distinct IRQs.

Even the current implementation by no means addresses it. Although it might seem a case for architectures to report the number of IRQs available for a driver to retry, in fact they all just fail. The same applies to *any* other type of resource involved: irq_desc's, CPU interrupt vector space, msi_desc's etc. No platform cares about it and just bails out once a constrain met (please correct me if I am wrong here). Given that Linux has been doing well even on embedded I think we should not change it.

The only exception to the above is pSeries platform which takes advantage of the current design (to implement MSI quota). There are indications we can satisfy pSeries requirements, but the design proposed in this RFC is not going to change drastically anyway. The start of the discusstion is here: https://lkml.org/lkml/2013/9/5/293

Quoted 3 lines

Currently pci_enable_msix() will report a positive value if it fails due to the global limit. Your patch 7 removes that. pci_enable_msi_block() unfortunately doesn't appear to do this.

pci_enable_msi_block() can do more than one MSI only on x86 (with IOMMU), but it does not bother to return positive numbers, indeed.

Quoted 5 lines

It seems to me that a more useful interface would take a minimum and maximum number of vectors from the driver. This wouldn't allow the driver to specify that it could only accept, say, any even number within a certain range, but you could still leave the current functions available for any driver that needs that.

Mmmm.. I am not sure I am getting it. Could you please rephrase?

Quoted 1 line

Ben.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#116 lore raw
Quoted 7 lines
Quoted 5 lines

It seems to me that a more useful interface would take a minimum and maximum number of vectors from the driver. This wouldn't allow the driver to specify that it could only accept, say, any even number within a certain range, but you could still leave the current functions available for any driver that needs that.

Mmmm.. I am not sure I am getting it. Could you please rephrase?

One possibility is for drivers than can use a lot of interrupts to request a minimum number initially and then request the additional ones much later on.
That would make it less likely that none will be available for devices/drivers that need them but are initialised later.

David

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

#117 lore raw
Quoted 7 lines · David Laight
Quoted 1 line

Mmmm.. I am not sure I am getting it. Could you please rephrase?

One possibility is for drivers than can use a lot of interrupts to request a minimum number initially and then request the additional ones much later on.
That would make it less likely that none will be available for devices/drivers that need them but are initialised later.

It sounds as a whole new topic for me. Isn't it?

Anyway, what prevents the above from being done with pci_enable_msix(nvec1) - pci_disable_msix() - pci_enable_msix(nvec2) where nvec1 < nvec2?

Quoted 1 line

David

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#118 lore raw
In reply to Alexander Gordeev
Quoted 20 lines · Alexander Gordeev
Quoted 10 lines · Ben Hutchings
Quoted 3 lines · Alexander Gordeev

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

[...]

I think this is fundamentally flawed: pci_msix_table_size() and pci_get_msi_cap() can only report the limits of the *device* (which the driver usually already knows), whereas MSI allocation can also be constrained due to *global* limits on the number of distinct IRQs.

Even the current implementation by no means addresses it. Although it might seem a case for architectures to report the number of IRQs available for a driver to retry, in fact they all just fail. The same applies to *any* other type of resource involved: irq_desc's, CPU interrupt vector space, msi_desc's etc. No platform cares about it and just bails out once a constrain met (please correct me if I am wrong here). Given that Linux has been doing well even on embedded I think we should not change it.

Quoted 5 lines

The only exception to the above is pSeries platform which takes advantage of the current design (to implement MSI quota). There are indications we can satisfy pSeries requirements, but the design proposed in this RFC is not going to change drastically anyway. The start of the discusstion is here: https://lkml.org/lkml/2013/9/5/293

All I can see there is that Tejun didn't think that the global limits and positive return values were implemented by any architecture. But you have a counter-example, so I'm not sure what your point is.

It has been quite a while since I saw this happen on x86. But I just checked on a test system running RHEL 5 i386 (Linux 2.6.18). If I ask for 16 MSI-X vectors on a device that supports 1024, the return value is 8, and indeed I can then successfully allocate 8.

Now that's going quite a way back, and it may be that global limits aren't a significant problem any more. With the x86_64 build of RHEL 5 on an identical system, I can allocate 16 or even 32, so this is apparently not a hardware limit in this case.

Quoted 7 lines
Quoted 3 lines

Currently pci_enable_msix() will report a positive value if it fails due to the global limit. Your patch 7 removes that. pci_enable_msi_block() unfortunately doesn't appear to do this.

pci_enable_msi_block() can do more than one MSI only on x86 (with IOMMU), but it does not bother to return positive numbers, indeed.

Quoted 7 lines
Quoted 5 lines

It seems to me that a more useful interface would take a minimum and maximum number of vectors from the driver. This wouldn't allow the driver to specify that it could only accept, say, any even number within a certain range, but you could still leave the current functions available for any driver that needs that.

Mmmm.. I am not sure I am getting it. Could you please rephrase?

Most drivers seem to either:
(a) require exactly a certain number of MSI vectors, or
(b) require a minimum number of MSI vectors, usually want to allocate more, and work with any number in between

We can support drivers in both classes by adding new allocation functions that allow specifying a minimum (required) and maximum (wanted) number of MSI vectors. Those in class (a) would just specify the same value for both. These new functions can take account of any global limit or allocation policy without any further changes to the drivers that use them.

The few drivers with more specific requirements would still need to implement the currently recommended loop, using the old allocation functions.

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#119 lore raw
Quoted 3 lines · Ben Hutchings

On Fri, 2013-10-04 at 10:29 +0200, Alexander Gordeev wrote:
All I can see there is that Tejun didn't think that the global limits and positive return values were implemented by any architecture.

I would say more than just that :) I picked few quotes which in my reading represent the guys positions:

Tejun Heo: "...do we really
care about possible partial success? This sort of interface is unnecessarily complex and actively harmful. It forces all users to wonder what could possibly happen and implement all sorts of nutty fallback logic which is highly likely to be error-prone on both the software and hardware side. Seriously, how much testing would such code path would get both on the driver and firmware sides?"

Bjorn Helgaas: "I agree, that would be much simpler.
I propose that you rework it that way, and at least find out what (if anything) would break if we do that."

Michael Ellerman: "I really think you're overstating the complexity here.
Functions typically return a boolean -> nothing to see here
This function returns a tristate value -> brain explosion!"; "All a lot of bother for no real gain IMHO."

Quoted 1 line

But you have a counter-example, so I'm not sure what your point is.

I concur with Tejun. I think we need to get rid of the loop.

As of the counter-example I think we could honour the pSeries quota by inroducing an interface to interrogate what you call global limits - pci_get_msix_limit(), i.e.:

	rc = pci_msix_table_size(pdev, nvec);
	if (rc < 0)
		return rc;

	nvec = min(rc, nvec);

	rc = pci_get_msix_limit(pdev, nvec);
	if (rc < 0)
		return rc;

	nvec = min(rc, nvec);

	for (i = 0; i < nvec; i++)
		msix_entry[i].entry = i;

	rc = pci_enable_msix(pdev, msix_entry, nvec);
	if (rc)
		return rc;

The latest state of those discussion is somewhere around Michael's:
"We could probably make that work." and Tejun's: "Are we talking about some limited number of device drivers here? Also, is the quota still necessary for machines in production today?"

So my point is - drivers should first obtain a number of MSIs they *can* get, then *derive* a number of MSIs the device is fine with and only then request that number. Not terribly different from memory or any other type of resource allocation ;)

Quoted 9 lines

It has been quite a while since I saw this happen on x86. But I just checked on a test system running RHEL 5 i386 (Linux 2.6.18). If I ask for 16 MSI-X vectors on a device that supports 1024, the return value is 8, and indeed I can then successfully allocate 8.

Now that's going quite a way back, and it may be that global limits aren't a significant problem any more. With the x86_64 build of RHEL 5 on an identical system, I can allocate 16 or even 32, so this is apparently not a hardware limit in this case.

Well, I do not know how to comment here. 2.6.18 has a significantly different codebase wrt MSIs. What about a recent version?

Quoted 11 lines

Most drivers seem to either:
(a) require exactly a certain number of MSI vectors, or
(b) require a minimum number of MSI vectors, usually want to allocate more, and work with any number in between

We can support drivers in both classes by adding new allocation functions that allow specifying a minimum (required) and maximum (wanted) number of MSI vectors. Those in class (a) would just specify the same value for both. These new functions can take account of any global limit or allocation policy without any further changes to the drivers that use them.

I think such interface is redundant wrt the current pci_enable_msix() implementation.. and you propose to leave it. IMO it unnecessarily blows the generic MSI API with no demand from drivers.

Quoted 3 lines

The few drivers with more specific requirements would still need to implement the currently recommended loop, using the old allocation functions.

Although the classes of drivers you specified indeed exist, I do believe just a single interface is enough to handle them all.

Quoted 1 line

Ben.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#83 lore raw

Re: [PATCH RFC 54/77] ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Jon Mason
Quoted 6 lines · Jon Mason
Quoted 5 lines · Alexander Gordeev
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 12 lines

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1069

Quoted 10 lines
drivers/ntb/ntb_hw.c +1 -1
index de2062c..eccd5e5 100644
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Nack, this check is unnecessary.

If SNB can do more than SNB_MSIX_CNT MSI-Xs then this check is needed to enable less than maximum MSI-Xs in case the maximum was not allocated. Otherwise SNB will fallback to single MSI instead of multiple MSI-Xs.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#120 lore raw
In reply to Alexander Gordeev
Quoted 4 lines · Alexander Gordeev

So my point is - drivers should first obtain a number of MSIs they *can* get, then *derive* a number of MSIs the device is fine with and only then request that number. Not terribly different from memory or any other type of resource allocation ;)

What if the limit is for a group of devices ? Your interface is racy in that case, another driver could have eaten into the limit in between the calls.

Ben.

#121 lore raw
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 4 lines · Alexander Gordeev

So my point is - drivers should first obtain a number of MSIs they *can* get, then *derive* a number of MSIs the device is fine with and only then request that number. Not terribly different from memory or any other type of resource allocation ;)

What if the limit is for a group of devices ? Your interface is racy in that case, another driver could have eaten into the limit in between the calls.

Well, the another driver has had a better karma ;) But seriously, the current scheme with a loop is not race-safe wrt to any other type of resource which might exhaust. What makes the quota so special so we should care about it and should not care i.e. about lack of msi_desc's?

Yeah, I know the quota might hit more likely. But why it is not addressed right now then? Not a single function in chains...
rtas_msi_check_device() -> msi_quota_for_device() -> traverse_pci_devices() rtas_setup_msi_irqs() -> msi_quota_for_device() -> traverse_pci_devices()
...is race-safe. So if it has not been bothering anyone until now then no reason to start worrying now :)

In fact, in the current design to address the quota race decently the drivers would have to protect the *loop* to prevent the quota change between a pci_enable_msix() returned a positive number and the the next call to pci_enable_msix() with that number. Is it doable?

Quoted 3 lines

Ben.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#122 lore raw
Quoted 7 lines · Alexander Gordeev
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 4 lines · Alexander Gordeev

So my point is - drivers should first obtain a number of MSIs they *can* get, then *derive* a number of MSIs the device is fine with and only then request that number. Not terribly different from memory or any other type of resource allocation ;)

Quoted 8 lines
Quoted 3 lines

What if the limit is for a group of devices ? Your interface is racy in that case, another driver could have eaten into the limit in between the calls.

Well, the another driver has had a better karma ;) But seriously, the current scheme with a loop is not race-safe wrt to any other type of resource which might exhaust. What makes the quota so special so we should care about it and should not care i.e. about lack of msi_desc's?

I'm not saying the current scheme is better but I prefer the option of passing a min,max to the request function.

Quoted 11 lines

Yeah, I know the quota might hit more likely. But why it is not addressed right now then? Not a single function in chains...
rtas_msi_check_device() -> msi_quota_for_device() -> traverse_pci_devices() rtas_setup_msi_irqs() -> msi_quota_for_device() -> traverse_pci_devices()
...is race-safe. So if it has not been bothering anyone until now then no reason to start worrying now :)

In fact, in the current design to address the quota race decently the drivers would have to protect the *loop* to prevent the quota change between a pci_enable_msix() returned a positive number and the the next call to pci_enable_msix() with that number. Is it doable?

I am not advocating for the current design, simply saying that your proposal doesn't address this issue while Ben's does.

Cheers,
Ben.

Quoted 4 lines
Quoted 3 lines

Ben.

#123 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Alexander Gordeev

In fact, in the current design to address the quota race decently the drivers would have to protect the *loop* to prevent the quota change between a pci_enable_msix() returned a positive number and the the next call to pci_enable_msix() with that number. Is it doable?

I am not advocating for the current design, simply saying that your proposal doesn't address this issue while Ben's does.

There is one major flaw in min-max approach - the generic MSI layer will have to take decisions on exact number of MSIs to request, not device drivers.

This will never work for all devices, because there might be specific requirements which are not covered by the min-max. That is what Ben described "...say, any even number within a certain range". Ben suggests to leave the existing loop scheme to cover such devices, which I think is not right.

What about introducing pci_lock_msi() and pci_unlock_msi() and let device drivers care about their ranges and specifics in race-safe manner? I do not call to introduce it right now (since it appears pSeries has not been hitting the race for years) just as a possible alternative to Ben's proposal.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#84 lore raw

Re: [PATCH RFC 54/77] ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Alexander Gordeev
Quoted 20 lines · Alexander Gordeev
Quoted 6 lines · Jon Mason
Quoted 5 lines · Alexander Gordeev
Signed-off-by Alexander Gordeev <agordeev at redhat.com>
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 12 lines

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1069

Quoted 10 lines
drivers/ntb/ntb_hw.c +1 -1
index de2062c..eccd5e5 100644
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Nack, this check is unnecessary.

Quoted 3 lines

If SNB can do more than SNB_MSIX_CNT MSI-Xs then this check is needed to enable less than maximum MSI-Xs in case the maximum was not allocated. Otherwise SNB will fallback to single MSI instead of multiple MSI-Xs.

Per the comment in the code snippet above, "If we can't get all 4, then we can't use MSI-X". There is already a check to see if more than 4 were acquired. So it's not possible to hit this. Even if it was, don't use SNB_MSIX_CNT here (limits.msix_cnt is the preferred variable). Also, the "()" are unnecessary.

Thanks,
Jon

Quoted 4 lines
Signature · 3 lines
Regards,
Alexander Gordeev
agordeev at redhat.com
#124 lore raw
In reply to Alexander Gordeev

Hey, guys.

Quoted 9 lines · Alexander Gordeev
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Alexander Gordeev

In fact, in the current design to address the quota race decently the drivers would have to protect the *loop* to prevent the quota change between a pci_enable_msix() returned a positive number and the the next call to pci_enable_msix() with that number. Is it doable?

I am not advocating for the current design, simply saying that your proposal doesn't address this issue while Ben's does.

Hmmm... yean, the race condition could be an issue as multiple msi allocation might fail even if the driver can and explicitly handle multiple allocation if the quota gets reduced inbetween.

Quoted 3 lines

There is one major flaw in min-max approach - the generic MSI layer will have to take decisions on exact number of MSIs to request, not device drivers.

The min-max approach would actually be pretty nice for the users which actually care about this.

Quoted 5 lines

This will never work for all devices, because there might be specific requirements which are not covered by the min-max. That is what Ben described "...say, any even number within a certain range". Ben suggests to leave the existing loop scheme to cover such devices, which I think is not right.

if it could work.

Quoted 5 lines

What about introducing pci_lock_msi() and pci_unlock_msi() and let device drivers care about their ranges and specifics in race-safe manner? I do not call to introduce it right now (since it appears pSeries has not been hitting the race for years) just as a possible alternative to Ben's proposal.

I don't think the same race condition would happen with the loop. The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

As long as the number of drivers which need this sort of adaptive allocation isn't too high and the common cases can be made simple, I don't think the "complex" part of interface is all that important. Maybe we can have reserve / cancel type interface or just keep the loop with more explicit function names (ie. try_enable or something like that).

Thanks.

Signature · 5 lines
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#12 lore raw

Re: [PATCH RFC 05/77] PCI/MSI: Convert pci_msix_table_size() to a public interface

In reply to Alexander Gordeev

Hello,

Quoted 9 lines · Alexander Gordeev

Make pci_msix_table_size() to return a error code if the device does not support MSI-X. This update is needed to facilitate a forthcoming re-design MSI/MSI-X interrupts enabling pattern.

Device drivers will use this interface to obtain maximum number of MSI-X interrupts the device supports and use that value in the following call to pci_enable_msix() interface.

Hmmm... I probably missed something but why is this necessary? To discern between -EINVAL and -ENOSPC? If so, does that really matter?

Thanks.

Signature · 5 lines
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#18 lore raw

Re: [PATCH RFC 07/77] PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern

In reply to Alexander Gordeev

Hello,

Documentation/PCI/MSI-HOWTO.txt the device are in the range dev->irq to dev->irq + count - 1. · line 134

Quoted 13 lines · Alexander Gordeev
Documentation/PCI/MSI-HOWTO.txt +13 -0
+static int foo_driver_enable_msi(struct foo_adapter *adapter, int nvec)
+{
+	rc = pci_get_msi_cap(adapter->pdev);
+	if (rc < 0)
+		return rc;
+
+	nvec = min(nvec, rc);
+	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
+		return -ENOSPC;
+
+	rc = pci_enable_msi_block(adapter->pdev, nvec);
+	return rc;
+}

If there are many which duplicate the above pattern, it'd probably be worthwhile to provide a helper? It's usually a good idea to reduce the amount of boilerplate code in drivers.

Quoted 6 lines

static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) {

Documentation/PCI/MSI-HOWTO.txt +4 -0
+	rc = pci_msix_table_size(adapter->pdev);
+	if (rc < 0)
+		return rc;
+

Documentation/PCI/MSI-HOWTO.txt number once MSI-X is enabled. · line 257

Quoted 10 lines
Documentation/PCI/MSI-HOWTO.txt +9 -0
+	nvec = min(nvec, rc);
+	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
+		return -ENOSPC;
+
+	for (i = 0; i < nvec; i++)
+		adapter->msix_entries[i].entry = i;
+
+	rc = pci_enable_msix(adapter->pdev, adapter->msix_entries, nvec);
+	return rc;
 }

Ditto.

drivers/pci/msi.c int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec) · line 957

Quoted 9 lines
drivers/pci/msi.c +1 -1
@@ -975,7 +951,7 @@ int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
 	if (nr_entries < 0)
 		return nr_entries;
 	if (nvec > nr_entries)
-		return nr_entries;
+		return -EINVAL;
 
 	/* Check for any invalid entries */
 	for (i = 0; i < nvec; i++) {

If we do things this way, it breaks all drivers using this interface until they're converted, right? Also, it probably isn't the best idea to flip the behavior like this as this can go completely unnoticed (no compiler warning or anything, the same function just behaves differently). Maybe it'd be a better idea to introduce a simpler interface that most can be converted to?

Thanks.

Signature · 1 line
tejun
#134 lore raw
In reply to Alexander Gordeev

Hello, Alexander.

Quoted 11 lines · Alexander Gordeev

Alexander Gordeev (77):
PCI/MSI: Fix return value when populate_msi_sysfs() failed PCI/MSI/PPC: Fix wrong RTAS error code reporting PCI/MSI/s390: Fix single MSI only check PCI/MSI/s390: Remove superfluous check of MSI type PCI/MSI: Convert pci_msix_table_size() to a public interface PCI/MSI: Factor out pci_get_msi_cap() interface PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern PCI/MSI: Get rid of pci_enable_msi_block_auto() interface
ahci: Update MSI/MSI-X interrupts enablement code
ahci: Check MRSM bit when multiple MSIs enabled

...

Whee.... that's a lot more than I expected. I was just scanning multiple msi users. Maybe we can stage the work in more manageable steps so that you don't have to go through massive conversion only to do it all over again afterwards and likewise people don't get bombarded on each iteration? Maybe we can first update pci / msi code proper, msi and then msix?

Thanks.

Signature · 1 line
tejun
#85 lore raw

Re: [PATCH RFC 54/77] ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Jon Mason
Quoted 25 lines · Jon Mason
Quoted 20 lines · Alexander Gordeev
Quoted 6 lines · Jon Mason
Quoted 5 lines · Alexander Gordeev
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 12 lines

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1069

Quoted 10 lines
drivers/ntb/ntb_hw.c +1 -1
index de2062c..eccd5e5 100644
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Nack, this check is unnecessary.

Quoted 3 lines

If SNB can do more than SNB_MSIX_CNT MSI-Xs then this check is needed to enable less than maximum MSI-Xs in case the maximum was not allocated. Otherwise SNB will fallback to single MSI instead of multiple MSI-Xs.

Quoted 5 lines

Per the comment in the code snippet above, "If we can't get all 4, then we can't use MSI-X". There is already a check to see if more than 4 were acquired. So it's not possible to hit this. Even if it was, don't use SNB_MSIX_CNT here (limits.msix_cnt is the preferred variable). Also, the "()" are unnecessary.

The changelog is definitely bogus. I meant here an improvement to the existing scheme, not a conversion to the new one:

	msix_entries = msix_table_size(val);

Getting i.e. 16 vectors here.

	if (msix_entries > ndev->limits.msix_cnt) {
		rc = -EINVAL;
		goto err;
	}

Upper limit check i.e. succeeds.

	[...]

	rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);

pci_enable_msix() does not success and returns i.e. 8 here, should retry.

	if (rc < 0)
		goto err1;
	if (rc > 0) {
		/* On SNB, the link interrupt is always tied to 4th vector.  If
		 * we can't get all 4, then we can't use MSI-X.
		 */
		if (ndev->hw_type != BWD_HW) {

On SNB bail out here, although could have continue with 8 vectors. Can only use SNB_MSIX_CNT here, since limits.msix_cnt is the upper limit.

			rc = -EIO;
			goto err1;
		}

		[...]
	}
Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#125 lore raw
In reply to Tejun Heo
Quoted 11 lines · Tejun Heo

I don't think the same race condition would happen with the loop. The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

As long as the number of drivers which need this sort of adaptive allocation isn't too high and the common cases can be made simple, I don't think the "complex" part of interface is all that important. Maybe we can have reserve / cancel type interface or just keep the loop with more explicit function names (ie. try_enable or something like that).

I'm thinking a better API overall might just have been to request individual MSI-X one by one :-)

We want to be able to request an MSI-X at runtime anyway ... if I want to dynamically add a queue to my network interface, I want it to be able to pop a new arbitrary MSI-X.

And we don't want to lock drivers into contiguous MSI-X sets either.

And for the cleanup ... well that's what the "pcim" functions are for, we can just make MSI-X variants.

Ben.

#86 lore raw

Re: [PATCH RFC 54/77] ntb: Ensure number of MSIs on SNB is enough for the link interrupt

In reply to Alexander Gordeev
Quoted 32 lines · Alexander Gordeev
Quoted 25 lines · Jon Mason
Quoted 20 lines · Alexander Gordeev
Quoted 6 lines · Jon Mason
Quoted 5 lines · Alexander Gordeev
Signed-off-by Alexander Gordeev <agordeev at redhat.com>
---
 drivers/ntb/ntb_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 12 lines

drivers/ntb/ntb_hw.c static int ntb_setup_msix(struct ntb_device *ndev) · line 1069

Quoted 10 lines
drivers/ntb/ntb_hw.c +1 -1
index de2062c..eccd5e5 100644
@@ -1066,7 +1066,7 @@ static int ntb_setup_msix(struct ntb_device *ndev)
 		/* On SNB, the link interrupt is always tied to 4th vector.  If
 		 * we can't get all 4, then we can't use MSI-X.
 		 */
-		if (ndev->hw_type != BWD_HW) {
+		if ((rc < SNB_MSIX_CNT) && (ndev->hw_type != BWD_HW)) {

Nack, this check is unnecessary.

Quoted 3 lines

If SNB can do more than SNB_MSIX_CNT MSI-Xs then this check is needed to enable less than maximum MSI-Xs in case the maximum was not allocated. Otherwise SNB will fallback to single MSI instead of multiple MSI-Xs.

Quoted 5 lines

Per the comment in the code snippet above, "If we can't get all 4, then we can't use MSI-X". There is already a check to see if more than 4 were acquired. So it's not possible to hit this. Even if it was, don't use SNB_MSIX_CNT here (limits.msix_cnt is the preferred variable). Also, the "()" are unnecessary.

Quoted 8 lines

The changelog is definitely bogus. I meant here an improvement to the existing scheme, not a conversion to the new one:

	msix_entries = msix_table_size(val);

Getting i.e. 16 vectors here.

	if (msix_entries > ndev->limits.msix_cnt) {

On SNB HW, limits.msix_cnt is set to SNB_MSIX_CNT (4)
http://lxr.free-electrons.com/source/drivers/ntb/ntb_hw.c#L558

Quoted 11 lines
		rc = -EINVAL;
		goto err;
	}

Upper limit check i.e. succeeds.

	[...]

	rc = pci_enable_msix(pdev, ndev->msix_entries, msix_entries);

pci_enable_msix() does not success and returns i.e. 8 here, should retry.

Per the above, since our upper bound is 4. We will either have this return 0 for all 4 or a number between 1 and 3 (or an error, but that's not relevant to this discussion).

Quoted 10 lines
	if (rc < 0)
		goto err1;
	if (rc > 0) {
		/* On SNB, the link interrupt is always tied to 4th vector.  If
		 * we can't get all 4, then we can't use MSI-X.
		 */
		if (ndev->hw_type != BWD_HW) {

On SNB bail out here, although could have continue with 8 vectors. Can only use SNB_MSIX_CNT here, since limits.msix_cnt is the upper limit.

Since we can guarantee that rc is between 1 and 3 at this point (on SNB HW), we should error out.

Thanks,
Jon

Quoted 12 lines
			rc = -EIO;
			goto err1;
		}

		[...]
	}
Signature · 3 lines
Regards,
Alexander Gordeev
agordeev at redhat.com
#126 lore raw
In reply to Benjamin Herrenschmidt
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 11 lines · Tejun Heo

I don't think the same race condition would happen with the loop. The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

As long as the number of drivers which need this sort of adaptive allocation isn't too high and the common cases can be made simple, I don't think the "complex" part of interface is all that important. Maybe we can have reserve / cancel type interface or just keep the loop with more explicit function names (ie. try_enable or something like that).

Quoted 6 lines

I'm thinking a better API overall might just have been to request individual MSI-X one by one :-)

We want to be able to request an MSI-X at runtime anyway ... if I want to dynamically add a queue to my network interface, I want it to be able to pop a new arbitrary MSI-X.

Yes, this would be very useful.

Quoted 1 line

And we don't want to lock drivers into contiguous MSI-X sets either.

I don't think there's any such limitation now. The entries array passed to pci_enable_msix() specifies which MSI-X vectors the driver wants to enable. It's usually filled with 0..nvec-1 in order, but not always. And the IRQ numbers returned aren't usually contiguous either, on x86.

Ben.

Quoted 2 lines

And for the cleanup ... well that's what the "pcim" functions are for, we can just make MSI-X variants.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#132 lore raw
In reply to Alexander Gordeev
Quoted 7 lines · Alexander Gordeev
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 4 lines · Alexander Gordeev

In fact, in the current design to address the quota race decently the drivers would have to protect the *loop* to prevent the quota change between a pci_enable_msix() returned a positive number and the the next call to pci_enable_msix() with that number. Is it doable?

Quoted 6 lines
Quoted 2 lines

I am not advocating for the current design, simply saying that your proposal doesn't address this issue while Ben's does.

There is one major flaw in min-max approach - the generic MSI layer will have to take decisions on exact number of MSIs to request, not device drivers.

[...

No, the min-max functions should be implemented using the same loop that drivers are expected to use now.

Ben.

Signature · 8 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#127 lore raw
In reply to Benjamin Herrenschmidt
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 11 lines · Tejun Heo

I don't think the same race condition would happen with the loop. The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

As long as the number of drivers which need this sort of adaptive allocation isn't too high and the common cases can be made simple, I don't think the "complex" part of interface is all that important. Maybe we can have reserve / cancel type interface or just keep the loop with more explicit function names (ie. try_enable or something like that).

Quoted 3 lines

We want to be able to request an MSI-X at runtime anyway ... if I want to dynamically add a queue to my network interface, I want it to be able to pop a new arbitrary MSI-X.

If you want to dynamically allocate another queue, you'd either need to have them all pre-allocated at alloc_etherdev_mqs(), or add a new API to netdev that allows adding new queues on the fly.

How things are done today, the Tx queues are all tacked onto the end of the netdev struct. That would have to change to probably a linked list of queues that could be grown or shrunk on the fly. netif_alloc_netdev_queues() would need to change the kzalloc() to a list allocation.

Cheers,
-PJ

#137 lore raw
In reply to Alexander Gordeev
Quoted 25 lines · Alexander Gordeev

This series is against "next" branch in Bjorn's repo:
git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git

Currently pci_enable_msi_block() and pci_enable_msix() interfaces return a error code in case of failure, 0 in case of success and a positive value which indicates the number of MSI-X/MSI interrupts that could have been allocated. The latter value should be passed to a repeated call to the interfaces until a failure or success:

	for (i = 0; i < FOO_DRIVER_MAXIMUM_NVEC; i++)
		adapter->msix_entries[i].entry = i;

	while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
		rc = pci_enable_msix(adapter->pdev,
				     adapter->msix_entries, nvec);
		if (rc > 0)
			nvec = rc;
		else
			return rc;
	}

	return -ENOSPC;
Quoted 2 lines

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

To clarify "Vast share of device drivers":

- 58 drivers call pci_enable_msix()
- 24 try a single allocation and then fallback to MSI/LSI
- 19 use the loop style allocation as above
- 14 try an allocation, and if it fails retry once
- 1 incorrectly continues when pci_enable_msix() returns > 0

So 33 drivers (> 50%) successfully make use of the "confusing and error-prone" return value.

Another 24 happily ignore it, which is also entirely fine.

And yes, one is buggy, so obviously the interface is too complex. Thanks drivers/ntb/ntb_hw.c

cheers

#138 lore raw
Quoted 6 lines · Michael Ellerman
Quoted 2 lines · Alexander Gordeev

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

To clarify "Vast share of device drivers":

Quoted 8 lines

- 58 drivers call pci_enable_msix()
- 24 try a single allocation and then fallback to MSI/LSI
- 19 use the loop style allocation as above
- 14 try an allocation, and if it fails retry once
- 1 incorrectly continues when pci_enable_msix() returns > 0

So 33 drivers (> 50%) successfully make use of the "confusing and error-prone" return value.

Ok, you caught me - 'vast share' is incorrect and is a subject to rewording. But out of 19/58 how many drivers tested fallbacks on the real hardware? IOW, which drivers are affected by the pSeries quota?

Quoted 2 lines

And yes, one is buggy, so obviously the interface is too complex. Thanks drivers/ntb/ntb_hw.c

vmxnet3 would be the best example.

Quoted 1 line

cheers

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#19 lore raw

Re: [PATCH RFC 07/77] PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern

In reply to Tejun Heo
Quoted 12 lines · Tejun Heo

Hello,

Quoted 9 lines · Alexander Gordeev
Documentation/PCI/MSI-HOWTO.txt +9 -0
+static int foo_driver_enable_msi(struct foo_adapter *adapter, int nvec)
+{
+	rc = pci_get_msi_cap(adapter->pdev);
+	if (rc < 0)
+		return rc;
+
+	nvec = min(nvec, rc);
+	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
+		return -ENOSPC;
Quoted 8 lines

Documentation/PCI/MSI-HOWTO.txt the device are in the range dev->irq to dev->irq + count - 1. · line 134

Quoted 4 lines
Documentation/PCI/MSI-HOWTO.txt +4 -0
+
+	rc = pci_enable_msi_block(adapter->pdev, nvec);
+	return rc;
+}

If there are many which duplicate the above pattern, it'd probably be worthwhile to provide a helper? It's usually a good idea to reduce the amount of boilerplate code in drivers.

I wanted to limit discussion in v1 to as little changes as possible. I 'planned' those helper(s) for a separate effort if/when the most important change is accepted and soaked a bit.

Quoted 7 lines
Quoted 7 lines
drivers/pci/msi.c +1 -1
@@ -975,7 +951,7 @@ int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
 	if (nr_entries < 0)
 		return nr_entries;
 	if (nvec > nr_entries)
-		return nr_entries;
+		return -EINVAL;
 
Quoted 5 lines

drivers/pci/msi.c int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec) · line 957

Quoted 2 lines
drivers/pci/msi.c +0 -0
 	/* Check for any invalid entries */
 	for (i = 0; i < nvec; i++) {

If we do things this way, it breaks all drivers using this interface until they're converted, right?

Right. And the rest of the series does it.

Quoted 5 lines

Also, it probably isn't the best idea
to flip the behavior like this as this can go completely unnoticed (no compiler warning or anything, the same function just behaves differently). Maybe it'd be a better idea to introduce a simpler interface that most can be converted to?

Well, an *other* interface is a good idea. What do you mean with the simpler here?

Quoted 1 line

tejun

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#13 lore raw

Re: [PATCH RFC 05/77] PCI/MSI: Convert pci_msix_table_size() to a public interface

In reply to Tejun Heo
Quoted 11 lines · Tejun Heo
Quoted 7 lines · Alexander Gordeev

Make pci_msix_table_size() to return a error code if the device does not support MSI-X. This update is needed to facilitate a forthcoming re-design MSI/MSI-X interrupts enabling pattern.

Device drivers will use this interface to obtain maximum number of MSI-X interrupts the device supports and use that value in the following call to pci_enable_msix() interface.

Hmmm... I probably missed something but why is this necessary? To discern between -EINVAL and -ENOSPC? If so, does that really matter?

pci_msix_table_size() is kind of helper and returns 0 if a device does not have MSI-X table. If we want drivers to use it we need return -EINVAL for MSI-X incapable/disabled devices. Nothing about -ENOSPC.

Quoted 1 line

tejun

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#135 lore raw
In reply to Tejun Heo
Quoted 6 lines · Tejun Heo

Whee.... that's a lot more than I expected. I was just scanning multiple msi users. Maybe we can stage the work in more manageable steps so that you don't have to go through massive conversion only to do it all over again afterwards and likewise people don't get bombarded on each iteration? Maybe we can first update pci / msi code proper, msi and then msix?

Multipe MSIs is just a handful of drivers, really. MSI-X impact still will be huge. But if we opt a different name for the new pci_enable_msix() then we could first update pci/msi, then drivers (in few stages possibly) and finally remove the old implementation.

Quoted 1 line

tejun

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#128 lore raw
In reply to Tejun Heo
Quoted 7 lines · Tejun Heo
Quoted 5 lines

What about introducing pci_lock_msi() and pci_unlock_msi() and let device drivers care about their ranges and specifics in race-safe manner? I do not call to introduce it right now (since it appears pSeries has not been hitting the race for years) just as a possible alternative to Ben's proposal.

I don't think the same race condition would happen with the loop.

We need to distinguish the contexts we're discussing.

If we talk about pSeries quota, then the current pSeries pci_enable_msix() implementation is racy internally and could fail if the quota went down *while* pci_enable_msix() is executing. In this case the loop will have to exit rather than retry with a lower number (what number?).

In this regard the new scheme does not bring anything new and relies on the fact this race does not hit and therefore does not worry.

If we talk about quota as it has to be, then yes - the loop scheme seems more preferable.

Overall, looks like we just need to fix the pSeries implementation, if the guys want it, he-he :)

Quoted 3 lines

The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

I am probably missing something here. If the global limit went down before inquiry then the inquiry will get what is available and try to allocate with than number.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#98 lore raw

RE: [PATCH RFC 63/77] qlcnic: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 15 lines

-----Original Message-----
From: Alexander Gordeev [mailto:[email protected]]
Sent: Wednesday, October 02, 2013 3:49 AM
To: linux-kernel
Cc: Alexander Gordeev; Bjorn Helgaas; Ralf Baechle; Michael Ellerman; Benjamin Herrenschmidt; Martin Schwidefsky; Ingo Molnar; Tejun Heo; Dan Williams; Andy King; Jon Mason; Matt Porter; linux-pci; linux-mips@linux- mips.org; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; netdev; [email protected]; Dept- Eng Linux Driver; Solarflare linux maintainers; VMware, Inc.; linux-scsi
Subject: [PATCH RFC 63/77] qlcnic: Update MSI/MSI-X interrupts enablement code

Quoted 4 lines

As result of recent re-design of the MSI/MSI-X interrupts enabling pattern this driver has to be updated to use the new technique to obtain a optimal number of MSI/MSI-X interrupts required.

"We will test this change for the driver and provide feedback."

Quoted 1 line

Thanks,
Himanshu

#139 lore raw
In reply to Alexander Gordeev
Quoted 13 lines · Alexander Gordeev
Quoted 12 lines · Michael Ellerman
Quoted 2 lines · Alexander Gordeev

This technique proved to be confusing and error-prone. Vast share of device drivers simply fail to follow the described guidelines.

To clarify "Vast share of device drivers":

- 58 drivers call pci_enable_msix()
- 24 try a single allocation and then fallback to MSI/LSI
- 19 use the loop style allocation as above
- 14 try an allocation, and if it fails retry once
- 1 incorrectly continues when pci_enable_msix() returns > 0

Quoted 6 lines
Quoted 2 lines

So 33 drivers (> 50%) successfully make use of the "confusing and error-prone" return value.

Ok, you caught me - 'vast share' is incorrect and is a subject to rewording. But out of 19/58 how many drivers tested fallbacks on the real hardware? IOW, which drivers are affected by the pSeries quota?

It's not 19/58, it's 33/58.

As to how many we care about on powerpc I can't say, so you have a point there. But I still think the interface is not actually that terrible.

cheers

#140 lore raw
In reply to Alexander Gordeev

On 13-10-02 06:29 AM, Alexander Gordeev wrote:
..

Quoted 3 lines

This update converts pci_enable_msix() and pci_enable_msi_block() interfaces to canonical kernel functions and makes them return a error code in case of failure or 0 in case of success.

Rather than silently break dozens of drivers in mysterious ways, please invent new function names for the replacements to the existing pci_enable_msix() and pci_enable_msi_block() functions.

That way, both in-tree and out-of-tree drivers will notice the API change, rather than having it go unseen and just failing for unknown reasons.

Thanks.

#141 lore raw
In reply to Alexander Gordeev
Quoted 8 lines · Alexander Gordeev

As result, device drivers will cease to use the overcomplicated repeated fallbacks technique and resort to a straightforward pattern - determine the number of MSI/MSI-X interrupts required before calling pci_enable_msi_block() and pci_enable_msix() interfaces:

Quoted 15 lines
	rc = pci_msix_table_size(adapter->pdev);
	if (rc < 0)
		return rc;

	nvec = min(nvec, rc);
	if (nvec < FOO_DRIVER_MINIMUM_NVEC) {
		return -ENOSPC;

	for (i = 0; i < nvec; i++)
		adapter->msix_entries[i].entry = i;

	rc = pci_enable_msix(adapter->pdev,
			     adapter->msix_entries, nvec);
	return rc;

Why not add a minimum number to pci_enable_msix(), i.e.:

pci_enable_msix(pdev, msix_entries, nvec, minvec)

... which means "nvec" is the number of interrupts *requested*, and "minvec" is the minimum acceptable number (otherwise fail).

-hpa

#142 lore raw
Quoted 6 lines · H. Peter Anvin

Why not add a minimum number to pci_enable_msix(), i.e.:

pci_enable_msix(pdev, msix_entries, nvec, minvec)

... which means "nvec" is the number of interrupts *requested*, and "minvec" is the minimum acceptable number (otherwise fail).

Which is exactly what Ben (the other Ben :-) suggested and that I supports...

Cheers,
Ben.

#130 lore raw
In reply to Tejun Heo
Quoted 3 lines · Tejun Heo

Hmmm... yean, the race condition could be an issue as multiple msi allocation might fail even if the driver can and explicitly handle multiple allocation if the quota gets reduced inbetween.

BTW, should we care about the quota getting increased inbetween? That would entail.. kind of pci_get_msi_limit() :), but IMHO it is not worth it.

Quoted 1 line

tejun

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#129 lore raw
In reply to Alexander Gordeev

Hello,

Quoted 4 lines · Alexander Gordeev

If we talk about pSeries quota, then the current pSeries pci_enable_msix() implementation is racy internally and could fail if the quota went down *while* pci_enable_msix() is executing. In this case the loop will have to exit rather than retry with a lower number (what number?).

Ah, okay, so that one is already broken.

Quoted 8 lines

In this regard the new scheme does not bring anything new and relies on the fact this race does not hit and therefore does not worry.

If we talk about quota as it has to be, then yes - the loop scheme seems more preferable.

Overall, looks like we just need to fix the pSeries implementation, if the guys want it, he-he :)

If we can't figure out a better interface for the retry case, I think what can really help is having a simple interface for the simpler cases.

Quoted 7 lines
Quoted 3 lines

The problem case is where multiple msi(x) allocation fails completely because the global limit went down before inquiry and allocation. In the loop based interface, it'd retry with the lower number.

I am probably missing something here. If the global limit went down before inquiry then the inquiry will get what is available and try to allocate with than number.

Oh, I should have written between inquiry and allocation. Sorry.

Thanks.

Signature · 1 line
tejun
#131 lore raw
In reply to Alexander Gordeev

Hello,

Quoted 8 lines · Alexander Gordeev
Quoted 3 lines · Tejun Heo

Hmmm... yean, the race condition could be an issue as multiple msi allocation might fail even if the driver can and explicitly handle multiple allocation if the quota gets reduced inbetween.

BTW, should we care about the quota getting increased inbetween? That would entail.. kind of pci_get_msi_limit() :), but IMHO it is not worth it.

I think we shouldn't. If the resource was low during a point in time during allocation, it's fine to base the result on that - the resource was actually low and which answer we return is just a question of timing and both are correct. The only reason the existing race condition is problematic is because it may fail even if the resource never falls below the failure point.

Thanks.

Signature · 1 line
tejun
#133 lore raw
In reply to Ben Hutchings
Quoted 7 lines · Ben Hutchings
Quoted 3 lines

There is one major flaw in min-max approach - the generic MSI layer will have to take decisions on exact number of MSIs to request, not device drivers.

[...

No, the min-max functions should be implemented using the same loop that drivers are expected to use now.

Wheee... earlier in the thread I thought you guys were referring to yourselves in the third person and was getting a bit worried. :)

Signature · 1 line
tejun
#20 lore raw

Re: [PATCH RFC 07/77] PCI/MSI: Re-design MSI/MSI-X interrupts enablement pattern

In reply to Alexander Gordeev

Hello, Alexander.

Quoted 7 lines · Alexander Gordeev
Quoted 3 lines

If there are many which duplicate the above pattern, it'd probably be worthwhile to provide a helper? It's usually a good idea to reduce the amount of boilerplate code in drivers.

I wanted to limit discussion in v1 to as little changes as possible. I 'planned' those helper(s) for a separate effort if/when the most important change is accepted and soaked a bit.

The thing is doing it this way generates more churns and noises. Once the simpler ones live behind a wrapper which can be built on the existing interface, we can have both reduced cost and more latitude on the complex cases.

Quoted 4 lines
Quoted 2 lines

If we do things this way, it breaks all drivers using this interface until they're converted, right?

Right. And the rest of the series does it.

Which breaks bisection which we shouldn't do.

Quoted 8 lines
Quoted 5 lines

Also, it probably isn't the best idea
to flip the behavior like this as this can go completely unnoticed (no compiler warning or anything, the same function just behaves differently). Maybe it'd be a better idea to introduce a simpler interface that most can be converted to?

Well, an *other* interface is a good idea. What do you mean with the simpler here?

I'm still talking about a simpler wrapper for common cases, which is the important part anyway.

Thanks.

Signature · 1 line
tejun
#136 lore raw
In reply to Alexander Gordeev

Hello,

Quoted 1 line · Alexander Gordeev

Multipe MSIs is just a handful of drivers, really. MSI-X impact still

Yes, so it's pretty nice to try out things there before going full-on.

Quoted 3 lines

will be huge. But if we opt a different name for the new pci_enable_msix() then we could first update pci/msi, then drivers (in few stages possibly) and finally remove the old implementation.

Yes, that probably should be the steps to follow eventually. My point was that you don't have to submit patches for all 7x conversions for an RFC round. Scanning them and seeing what would be necessary definitely is a good idea but just giving summary of the full conversion with several examples should be good enough before settling on the way forward, which should be easier for all involved.

Thanks a lot for your effort!

Signature · 1 line
tejun
#143 lore raw
In reply to Benjamin Herrenschmidt
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 6 lines · H. Peter Anvin

Why not add a minimum number to pci_enable_msix(), i.e.:

pci_enable_msix(pdev, msix_entries, nvec, minvec)

... which means "nvec" is the number of interrupts *requested*, and "minvec" is the minimum acceptable number (otherwise fail).

Which is exactly what Ben (the other Ben :-) suggested and that I supports...

Ok, this suggestion sounded in one or another form by several people. What about name it pcim_enable_msix_range() and wrap in couple more helpers to complete an API:

int pcim_enable_msix_range(pdev, msix_entries, nvec, minvec);
<0 - error code
>0 - number of MSIs allocated, where minvec >= result <= nvec

int pcim_enable_msix(pdev, msix_entries, nvec);
<0 - error code
>0 - number of MSIs allocated, where 1 >= result <= nvec

int pcim_enable_msix_exact(pdev, msix_entries, nvec);
	<0 - error code
	>0 - number of MSIs allocated, where result == nvec

The latter's return value seems odd, but I can not help to make it consistent with the first two.

(Sorry if you see this message twice - my MUA seems struggle with one of CC).

Quoted 4 lines

Cheers,
Ben.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#75 lore raw

Re: [PATCH RFC 50/77] mlx5: Update MSI/MSI-X interrupts enablement code

In reply to Alexander Gordeev
Quoted 3 lines · Alexander Gordeev

pci_enable_msix() may fail, but it can not return a positive number.

That is true according to the current logic but the comment on top of pci_enable_msix() still says:
"A return of < 0 indicates a failure. Or a return of > 0 indicates that driver request is exceeding the number of irqs or MSI-X vectors available"

So you're counting on an implementation that may change in the future. I think leaving the code as it is now is safer.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#144 lore raw
In reply to Alexander Gordeev
Quoted 14 lines · Alexander Gordeev

On Wed, Oct 09, 2013 at 03:24:08PM +1100, Benjamin Herrenschmidt wrote:

Ok, this suggestion sounded in one or another form by several people. What about name it pcim_enable_msix_range() and wrap in couple more helpers to complete an API:

int pcim_enable_msix_range(pdev, msix_entries, nvec, minvec);
<0 - error code
>0 - number of MSIs allocated, where minvec >= result <= nvec

int pcim_enable_msix(pdev, msix_entries, nvec);
<0 - error code
>0 - number of MSIs allocated, where 1 >= result <= nvec

Quoted 7 lines
int pcim_enable_msix_exact(pdev, msix_entries, nvec);
	<0 - error code
	>0 - number of MSIs allocated, where result == nvec

The latter's return value seems odd, but I can not help to make it consistent with the first two.

Is there a reason for the wrappers, as opposed to just specifying either 1 or nvec as the minimum?

-hpa

#145 lore raw
Quoted 19 lines · H. Peter Anvin
Quoted 18 lines · Alexander Gordeev

On Wed, Oct 09, 2013 at 03:24:08PM +1100, Benjamin Herrenschmidt wrote:

Ok, this suggestion sounded in one or another form by several people. What about name it pcim_enable_msix_range() and wrap in couple more helpers to complete an API:

int pcim_enable_msix_range(pdev, msix_entries, nvec, minvec);
<0 - error code
>0 - number of MSIs allocated, where minvec >= result <= nvec

int pcim_enable_msix(pdev, msix_entries, nvec);
<0 - error code
>0 - number of MSIs allocated, where 1 >= result <= nvec

int pcim_enable_msix_exact(pdev, msix_entries, nvec);
	<0 - error code
	>0 - number of MSIs allocated, where result == nvec
Quoted 6 lines
Quoted 3 lines

The latter's return value seems odd, but I can not help to make it consistent with the first two.

Is there a reason for the wrappers, as opposed to just specifying either 1 or nvec as the minimum?

The wrappers are more handy IMO.

I.e. can imagine people start struggling to figure out what minvec to provide:
1 or 0? Why 1? Oh.. okay.. Or should we tolerate 0 (as opposite to -ERANGE)?

Well, do not know.. pcim_enable_msix(pdev, msix_entries, nvec, nvec) is less readable for me than just pcim_enable_msix_exact(pdev, msix_entries, nvec).

Quoted 1 line

-hpa

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#146 lore raw

Just to help us all understand "the loop" issue..

Here's an example of driver code which uses the existing MSI-X interfaces, for a device which can work with either 16, 8, 4, 2, or 1 MSI-X interrupt. This is from a new driver I'm working on right now:

static int xx_alloc_msix_irqs (struct xx_dev *dev, int nvec)

{
        xx_disable_all_irqs(dev);
        do {
                if (nvec < 2)
                        xx_prep_for_1_msix_vector(dev);
                else if (nvec < 4)
                        xx_prep_for_2_msix_vectors(dev);
                else if (nvec < 8)
                        xx_prep_for_4_msix_vectors(dev);
                else if (nvec < 16)
                        xx_prep_for_8_msix_vectors(dev);
                else
                        xx_prep_for_16_msix_vectors(dev);
                nvec = pci_enable_msix(dev->pdev, dev->irqs, dev->num_vectors);
        } while (nvec > 0);

        if (nvec) {
                kerr(dev->name, "pci_enable_msix() failed, err=%d", nvec);
                dev->num_vectors = 0;
                return nvec;
        }
        return 0;       /* success */
}
#147 lore raw
Quoted 24 lines · Mark Lord

Just to help us all understand "the loop" issue..

Here's an example of driver code which uses the existing MSI-X interfaces, for a device which can work with either 16, 8, 4, 2, or 1 MSI-X interrupt. This is from a new driver I'm working on right now:

static int xx_alloc_msix_irqs (struct xx_dev *dev, int nvec)

{
        xx_disable_all_irqs(dev);
        do {
                if (nvec < 2)
                        xx_prep_for_1_msix_vector(dev);
                else if (nvec < 4)
                        xx_prep_for_2_msix_vectors(dev);
                else if (nvec < 8)
                        xx_prep_for_4_msix_vectors(dev);
                else if (nvec < 16)
                        xx_prep_for_8_msix_vectors(dev);
                else
                        xx_prep_for_16_msix_vectors(dev);
                nvec = pci_enable_msix(dev->pdev, dev->irqs, dev->num_vectors);
        } while (nvec > 0);
Quoted 7 lines
        if (nvec) {
                kerr(dev->name, "pci_enable_msix() failed, err=%d", nvec);
                dev->num_vectors = 0;
                return nvec;
        }
        return 0;       /* success */
}

Yeah, that is a very good example.

I would move all xx_prep_for_<pow2>_msix_vector() functions to a single helper i.e. xx_prep_msix_vectors(dev, ndev).

Considering also (a) we do not want to waste unused platform resources associated with MSI-Xs and pull more quota than needed and (b) fixing couple of bugs, this function could look like this:

static int xx_alloc_msix_irqs(struct xx_dev *dev, int nvec_in)

{
	int nvec = roundup_pow_of_two(nvec_in);	/* assume 0 > nvec_in <= 16 */
	int rc;

	xx_disable_all_irqs(dev);

retry:

       	xx_prep_for_msix_vectors(dev, nvec);

	rc = pci_enable_msix(dev->pdev, dev->irqs, nvec);	/* (b) */
	if (rc > 0) {
		nvec = rounddown_pow_of_two(nvec);		/* (a) */
		goto retry;
	}

	if (rc) {
		kerr(dev->name, "pci_enable_msix() failed, err=%d", rc);
		dev->num_vectors = 0;
		return rc;
	}

	dev->num_vectors = nvec;				/* (b) */
	return 0;       /* success */
}

Now, this is a loop-free alternative:

static int xx_alloc_msix_irqs(struct xx_dev *dev, int nvec)

{
	nvec = roundup_pow_of_two(nvec);	/* assume 0 > nvec <= 16 */

	xx_disable_all_irqs(dev);

	pci_lock_msi(dev->pdev);

	rc = pci_get_msix_limit(dev->pdev, nvec);
	if (rc < 0)
		goto err;

	nvec = min(nvec, rc);		/* if limit is more than requested */
	nvec = rounddown_pow_of_two(nvec);	/* (a) */

	xx_prep_for_msix_vectors(dev, nvec);

	rc = pci_enable_msix(dev->pdev, dev->irqs, nvec);	/* (b)	*/
	if (rc < 0)
		goto err;

	pci_unlock_msi(dev->pdev);

	dev->num_vectors = nvec;		/* (b) */
	return 0;

err:

	pci_unlock_msi(dev->pdev);

        kerr(dev->name, "pci_enable_msix() failed, err=%d", rc);
        dev->num_vectors = 0;
        return rc;
}
Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#148 lore raw
Quoted 6 lines · Alexander Gordeev
Quoted 5 lines · Mark Lord

Just to help us all understand "the loop" issue..

Here's an example of driver code which uses the existing MSI-X interfaces, for a device which can work with either 16, 8, 4, 2, or 1 MSI-X interrupt. This is from a new driver I'm working on right now:

..

Quoted 28 lines

Now, this is a loop-free alternative:

static int xx_alloc_msix_irqs(struct xx_dev *dev, int nvec)

{
	nvec = roundup_pow_of_two(nvec);	/* assume 0 > nvec <= 16 */

	xx_disable_all_irqs(dev);

	pci_lock_msi(dev->pdev);

	rc = pci_get_msix_limit(dev->pdev, nvec);
	if (rc < 0)
		goto err;

	nvec = min(nvec, rc);		/* if limit is more than requested */
	nvec = rounddown_pow_of_two(nvec);	/* (a) */

	xx_prep_for_msix_vectors(dev, nvec);

	rc = pci_enable_msix(dev->pdev, dev->irqs, nvec);	/* (b)	*/
	if (rc < 0)
		goto err;

	pci_unlock_msi(dev->pdev);

	dev->num_vectors = nvec;		/* (b) */
	return 0;
Quoted 7 lines

err:

	pci_unlock_msi(dev->pdev);

        kerr(dev->name, "pci_enable_msix() failed, err=%d", rc);
        dev->num_vectors = 0;
        return rc;
}

That would still need a loop, to handle the natural race between
the calls to pci_get_msix_limit() and pci_enable_msix() -- the driver and device can and should fall back to a smaller number of vectors when pci_enable_msix() fails.

#149 lore raw
Quoted 29 lines · Mark Lord
Quoted 29 lines

static int xx_alloc_msix_irqs(struct xx_dev *dev, int nvec)

{
	nvec = roundup_pow_of_two(nvec);	/* assume 0 > nvec <= 16 */

	xx_disable_all_irqs(dev);

	pci_lock_msi(dev->pdev);

	rc = pci_get_msix_limit(dev->pdev, nvec);
	if (rc < 0)
		goto err;

	nvec = min(nvec, rc);		/* if limit is more than requested */
	nvec = rounddown_pow_of_two(nvec);	/* (a) */

	xx_prep_for_msix_vectors(dev, nvec);

	rc = pci_enable_msix(dev->pdev, dev->irqs, nvec);	/* (b)	*/
	if (rc < 0)
		goto err;

	pci_unlock_msi(dev->pdev);

	dev->num_vectors = nvec;		/* (b) */
	return 0;

err:

	pci_unlock_msi(dev->pdev);
Quoted 8 lines
Quoted 4 lines
        kerr(dev->name, "pci_enable_msix() failed, err=%d", rc);
        dev->num_vectors = 0;
        return rc;
}

That would still need a loop, to handle the natural race between
the calls to pci_get_msix_limit() and pci_enable_msix() -- the driver and device can and should fall back to a smaller number of vectors when pci_enable_msix() fails.

Could you please explain why the value returned by pci_get_msix_limit() might change before pci_enable_msix() returned, while both protected by pci_lock_msi()?

Anyway, although the loop-free code (IMHO) reads better, pci_lock_msi() it is not a part of the original proposal and the more I think about it the less I like it.

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]