Skip to contentSkip to conversation

PCI/MSI: Allocate as many multiple-MSIs as requested

PATCH In review Alexander Gordeev 4 participants 10 messages May 13, 2013 – Jun 6, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover PCI/MSI: Allocate as many multiple-MSIs as requested

This series is against tip's x86/apic branch.

Allow conserving interrupt resources when PCI devices in multiple-MSI mode send a number of MSIs which is not a power-of-two. This update will prevent wasting of any resources associated with unused MSIs in general (i.e. IRQ descriptors) and x86 interrupt vectors in particular (the latter are notoriously scarce).

Recently PLX Technology confirmed they do have a relevant hardware, i.e. their new PEX8796 chip can send 18 MSIs.

Changes from v2:
- patch 2/2 changelog message elaborated

Changes from v1:
- do not conserve on IRTEs to prevent possible security hole when one device accesses other device's IRTEs

Patch 1 is a change to the generic MSI code
Patch 2 is the x86 enablement

Alexander Gordeev (2):
PCI/MSI: Allocate as many multiple-MSIs as requested x86/MSI: Conserve interrupt resources when using multiple-MSIs

 drivers/iommu/irq_remapping.c |   12 +++++++-----
 drivers/pci/msi.c             |   10 ++++++++--
 include/linux/msi.h           |    1 +
 3 files changed, 16 insertions(+), 7 deletions(-)
Signature · 7 lines
1.7.7.6


-- 
Regards,
Alexander Gordeev
[email protected]
#1 lore raw

PATCH v3 1/2 PCI/MSI: Allocate as many multiple-MSIs as requested

When multiple MSIs are enabled with pci_enable_msi_block(), the requested number of interrupts 'nvec' is rounded up to the nearest power-of-two value. The result is then used for setting up the number of MSI messages in the PCI device and allocation of interrupt resources in the operating system (i.e. vector numbers). Thus, in cases when a device driver requests some number of MSIs and this number is not a power-of-two value, the extra operating system resources (allocated as the result of rounding) are wasted.

This fix introduces 'msi_desc::nvec' field to address the above issue. When non-zero, it will report the actual number of MSIs the device will send, as requested by the device driver. This value should be used by architectures to properly set up and tear down associated interrupt resources.

Note, although the existing 'msi_desc::multiple' field might seem redundant, in fact in does not. In general case the number of MSIs a PCI device is initialized with is not necessarily the closest power- of-two value of the number of MSIs the device will send. Thus, in theory it would not be always possible to derive the former from the latter and we need to keep them both, to stress this corner case. Besides, since 'msi_desc::multiple' is a bitfield, throwing it out would not save us any space.

---
 drivers/pci/msi.c   |   10 ++++++++--
 include/linux/msi.h |    1 +
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/pci/msi.c +8 -2
index 00cc78c7..014b9d5 100644
@@ -79,7 +79,10 @@ void default_teardown_msi_irqs(struct pci_dev *dev)
 		int i, nvec;
 		if (entry->irq == 0)
 			continue;
-		nvec = 1 << entry->msi_attrib.multiple;
+		if (entry->nvec)
+			nvec = entry->nvec;
+		else
+			nvec = 1 << entry->msi_attrib.multiple;
 		for (i = 0; i < nvec; i++)
 			arch_teardown_msi_irq(entry->irq + i);
 	}
@@ -340,7 +343,10 @@ static void free_msi_irqs(struct pci_dev *dev)
 		int i, nvec;
 		if (!entry->irq)
 			continue;
-		nvec = 1 << entry->msi_attrib.multiple;
+		if (entry->nvec)
+			nvec = entry->nvec;
+		else
+			nvec = 1 << entry->msi_attrib.multiple;
 #ifdef CONFIG_GENERIC_HARDIRQS
 		for (i = 0; i < nvec; i++)
 			BUG_ON(irq_has_action(entry->irq + i));
include/linux/msi.h +1 -0
index ce93a34..0e20dfc 100644
@@ -35,6 +35,7 @@ struct msi_desc {
 
 	u32 masked;			/* mask bits */
 	unsigned int irq;
+	unsigned int nvec;		/* number of messages */
 	struct list_head list;
 
 	union {
Signature · 7 lines
1.7.7.6


-- 
Regards,
Alexander Gordeev
[email protected]
#8 lore raw

PATCH v3 2/2 x86/MSI: Conserve interrupt resources when using multiple-MSIs

In reply to Alexander Gordeev

Current multiple-MSI implementation does not take into account actual number of requested MSIs and always rounds that number to a closest power-of-two value. Yet, a number of MSIs a PCI device could send (and therefore a number of messages a device driver could request) may be a lesser power-of-two. As result, resources allocated for extra MSIs just wasted.

This update takes advantage of 'msi_desc::nvec' field introduced with generic MSI code to track number of requested and used MSIs. As result, resources associated with interrupts are conserved. Of those resources most noticeable are x86 interrupt vectors.

The initial version of this fix also consumed on IRTEs, but Jan noticed that a malfunctioning PCI device might send a message number it did not claim and thus refer an IRTE it does not own. To avoid this security hole the old-way approach preserved and as many IRTEs are reserved as the device could possibly send.

---
 drivers/iommu/irq_remapping.c |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/irq_remapping.c +7 -5
index d56f8c1..9eeb6cf 100644
@@ -51,26 +51,27 @@ static void irq_remapping_disable_io_apic(void)
 
 static int do_setup_msi_irqs(struct pci_dev *dev, int nvec)
 {
-	int node, ret, sub_handle, index = 0;
+	int node, ret, sub_handle, nvec_pow2, index = 0;
 	unsigned int irq;
 	struct msi_desc *msidesc;
 
-	nvec = __roundup_pow_of_two(nvec);
-
 	WARN_ON(!list_is_singular(&dev->msi_list));
 	msidesc = list_entry(dev->msi_list.next, struct msi_desc, list);
 	WARN_ON(msidesc->irq);
 	WARN_ON(msidesc->msi_attrib.multiple);
+	WARN_ON(msidesc->nvec);
 
 	node = dev_to_node(&dev->dev);
 	irq = __create_irqs(get_nr_irqs_gsi(), nvec, node);
 	if (irq == 0)
 		return -ENOSPC;
 
-	msidesc->msi_attrib.multiple = ilog2(nvec);
+	nvec_pow2 = __roundup_pow_of_two(nvec);
+	msidesc->nvec = nvec;
+	msidesc->msi_attrib.multiple = ilog2(nvec_pow2);
 	for (sub_handle = 0; sub_handle < nvec; sub_handle++) {
 		if (!sub_handle) {
-			index = msi_alloc_remapped_irq(dev, irq, nvec);
+			index = msi_alloc_remapped_irq(dev, irq, nvec_pow2);
 			if (index < 0) {
 				ret = index;
 				goto error;
@@ -95,6 +96,7 @@ error:
 	 * IRQs from tearing down again in default_teardown_msi_irqs()
 	 */
 	msidesc->irq = 0;
+	msidesc->nvec = 0;
 	msidesc->msi_attrib.multiple = 0;
 
 	return ret;
Signature · 7 lines
1.7.7.6


-- 
Regards,
Alexander Gordeev
[email protected]
#2 lore raw
In reply to Alexander Gordeev
Quoted 24 lines · Alexander Gordeev

When multiple MSIs are enabled with pci_enable_msi_block(), the requested number of interrupts 'nvec' is rounded up to the nearest power-of-two value. The result is then used for setting up the number of MSI messages in the PCI device and allocation of interrupt resources in the operating system (i.e. vector numbers). Thus, in cases when a device driver requests some number of MSIs and this number is not a power-of-two value, the extra operating system resources (allocated as the result of rounding) are wasted.

This fix introduces 'msi_desc::nvec' field to address the above issue. When non-zero, it will report the actual number of MSIs the device will send, as requested by the device driver. This value should be used by architectures to properly set up and tear down associated interrupt resources.

Note, although the existing 'msi_desc::multiple' field might seem redundant, in fact in does not. In general case the number of MSIs a PCI device is initialized with is not necessarily the closest power- of-two value of the number of MSIs the device will send. Thus, in theory it would not be always possible to derive the former from the latter and we need to keep them both, to stress this corner case. Besides, since 'msi_desc::multiple' is a bitfield, throwing it out would not save us any space.

Quoted 1 line

Would be nice to have an Acked-by from Bjorn for this patch.

Thanks,

Ingo

#9 lore raw

Re: [PATCH v3 -tip x86/apic 2/2] x86/MSI: Conserve interrupt resources when using multiple-MSIs

In reply to Alexander Gordeev
Quoted 12 lines · Alexander Gordeev

Current multiple-MSI implementation does not take into account actual number of requested MSIs and always rounds that number to a closest power-of-two value. Yet, a number of MSIs a PCI device could send (and therefore a number of messages a device driver could request) may be a lesser power-of-two. As result, resources allocated for extra MSIs just wasted.

This update takes advantage of 'msi_desc::nvec' field introduced with generic MSI code to track number of requested and used MSIs. As result, resources associated with interrupts are conserved. Of those resources most noticeable are x86 interrupt vectors.

Quoted 7 lines

The initial version of this fix also consumed on IRTEs, but Jan noticed that a malfunctioning PCI device might send a message number it did not claim and thus refer an IRTE it does not own. To avoid this security hole the old-way approach preserved and as many IRTEs are reserved as the device could possibly send.

This does not look all that bad.

Sebastian

#3 lore raw
In reply to Alexander Gordeev
Quoted 8 lines · Alexander Gordeev

Note, although the existing 'msi_desc::multiple' field might seem redundant, in fact in does not. In general case the number of MSIs a PCI device is initialized with is not necessarily the closest power- of-two value of the number of MSIs the device will send. Thus, in theory it would not be always possible to derive the former from the latter and we need to keep them both, to stress this corner case. Besides, since 'msi_desc::multiple' is a bitfield, throwing it out would not save us any space.

The last paragraph makes me curious. The only place where 'multiple' is set is in do_setup_msi_irqs() and this uses the next power of two for it. And since a device is not enabled twice, it is not overridden.
So it should be possible to compute 'multiple' out of 'nvec' but it saves cycles not do to so. I agree to keep 'multiple' but your argument does not seem to make sense.
While nitpicking, 'nvec' might deserve a better comment than 'number of messages' since it holds the number of allocated interrupts. :)

Sebastian

maintainer
#4 lore raw
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 8 lines · Alexander Gordeev

Note, although the existing 'msi_desc::multiple' field might seem redundant, in fact in does not. In general case the number of MSIs a PCI device is initialized with is not necessarily the closest power- of-two value of the number of MSIs the device will send. Thus, in theory it would not be always possible to derive the former from the latter and we need to keep them both, to stress this corner case. Besides, since 'msi_desc::multiple' is a bitfield, throwing it out would not save us any space.

Quoted 6 lines

The last paragraph makes me curious. The only place where 'multiple' is set is in do_setup_msi_irqs() and this uses the next power of two for it. And since a device is not enabled twice, it is not overridden.
So it should be possible to compute 'multiple' out of 'nvec' but it saves cycles not do to so. I agree to keep 'multiple' but your argument does not seem to make sense.

Alexander had an example device that advertised 16 vectors, but the driver knew that it could only generate 6. That's a case where we can't compute 'multiple' from 'nvec' (assuming the driver supplies 'nvec == 6'). If we just rounded up to compute 'multiple', I think we'd compute 8 instead of 16.

Quoted 2 lines

While nitpicking, 'nvec' might deserve a better comment than 'number of messages' since it holds the number of allocated interrupts. :)

I did change the name 'nvec' to 'nvec_used', which should help a bit. But I agree that it's still somewhat confusing.

BTW, the patches actually in my tree are at
http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=pci/alexander-msi
(I tweaked this name and some comments slightly).

Bjorn

#5 lore raw

-Suresh

Quoted 6 lines · Bjorn Helgaas

Alexander had an example device that advertised 16 vectors, but the driver knew that it could only generate 6. That's a case where we can't compute 'multiple' from 'nvec' (assuming the driver supplies 'nvec == 6'). If we just rounded up to compute 'multiple', I think we'd compute 8 instead of 16.

Sure, but as I said: the only place where 'multiple' is computed / written it is doing the round-up thingy.

Quoted 9 lines
Quoted 2 lines

While nitpicking, 'nvec' might deserve a better comment than 'number of messages' since it holds the number of allocated interrupts. :)

I did change the name 'nvec' to 'nvec_used', which should help a bit. But I agree that it's still somewhat confusing.

BTW, the patches actually in my tree are at
http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=pci/alexander-msi
(I tweaked this name and some comments slightly).

'nvec_used' is better the comment next to it is still wrong I think.

Quoted 1 line

Bjorn

Sebastian

#6 lore raw
In reply to Sebastian Andrzej Siewior
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 8 lines · Alexander Gordeev

Note, although the existing 'msi_desc::multiple' field might seem redundant, in fact in does not. In general case the number of MSIs a PCI device is initialized with is not necessarily the closest power- of-two value of the number of MSIs the device will send. Thus, in theory it would not be always possible to derive the former from the latter and we need to keep them both, to stress this corner case. Besides, since 'msi_desc::multiple' is a bitfield, throwing it out would not save us any space.

Quoted 8 lines

The last paragraph makes me curious. The only place where 'multiple' is set is in do_setup_msi_irqs() and this uses the next power of two for it. And since a device is not enabled twice, it is not overridden.
So it should be possible to compute 'multiple' out of 'nvec' but it saves cycles not do to so. I agree to keep 'multiple' but your argument does not seem to make sense.
While nitpicking, 'nvec' might deserve a better comment than 'number of messages' since it holds the number of allocated interrupts. :)

Sebastian,

I re-read my comment few times and I admit it might be confusing. You are right - 'multiple' is set by rounding up only. The part '...not necessarily the closest power-of-two value...' implied an abstract PCI device rather than the described code, but the wording is less than perfect, indeed.

In fact, at the moment of writing I kept in mind a follow-up patch that could help with aforementioned devices. That would be a new interface:

int pci_enable_msi_block_partial(struct pci_dev *dev,
unsigned int nvec_use, unsigned int nvec_init);

In this case 'nvec_use' would go to 'msi_desc::nvec_used' and 'nvec_init' would translate to 'msi_desc::multiple' in case 'nvec_init' is not zero. In case 'nvec_init' is zero, 'msi_desc::multiple' would be initialized with the maximum possible value for the device (the way it is done now for pci_enable_msi_block_auto() interface). So, for the AHCI device (Bjorn mentioned) such a call would conserve on 10 of 16 vectors:

	pci_enable_msi_block_partial(pdev, 6, 0);

What I am not sure is whether we need to read out the maximum possible number of vectors like pci_enable_msi_block_auto() does:

int pci_enable_msi_block_partial(struct pci_dev *dev,
unsigned int nvec_use, unsigned int nvec_init, unsigned int *maxvec);

I can not think of any use of 'maxvec' with this interface, but the second variant completes the whole picture about a device...

Quoted 1 line

Sebastian

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#7 lore raw
Quoted 1 line · Alexander Gordeev

Sebastian,

Hi Alexander,

Quoted 4 lines

I re-read my comment few times and I admit it might be confusing. You are right - 'multiple' is set by rounding up only. The part '...not necessarily the closest power-of-two value...' implied an abstract PCI device rather than the described code, but the wording is less than perfect, indeed.

Good, so it is not just me :)

Quoted 7 lines

In fact, at the moment of writing I kept in mind a follow-up patch that could help with aforementioned devices. That would be a new interface:

int pci_enable_msi_block_partial(struct pci_dev *dev,
unsigned int nvec_use, unsigned int nvec_init);

Quoted 8 lines

In this case 'nvec_use' would go to 'msi_desc::nvec_used' and 'nvec_init' would translate to 'msi_desc::multiple' in case 'nvec_init' is not zero. In case 'nvec_init' is zero, 'msi_desc::multiple' would be initialized with the maximum possible value for the device (the way it is done now for pci_enable_msi_block_auto() interface). So, for the AHCI device (Bjorn mentioned) such a call would conserve on 10 of 16 vectors:

	pci_enable_msi_block_partial(pdev, 6, 0);

Ah okay. that makes sense.

Quoted 11 lines

What I am not sure is whether we need to read out the maximum possible number of vectors like pci_enable_msi_block_auto() does:

int pci_enable_msi_block_partial(struct pci_dev *dev,
unsigned int nvec_use, unsigned int nvec_init, unsigned int *maxvec);

I can not think of any use of 'maxvec' with this interface, but the second variant completes the whole picture about a device...

The user of pci_enable_msi_block_auto() does not know how many it will get so argument seems essential. Your new function on the other hand says exactly how many it requires. Anything less should be an error.

Sebastian