Skip to contentSkip to conversation

msi-x: let drivers retry when not enough vectors

PATCH Applied Michael S. Tsirkin 8 participants 34 messages May 7, 2009 – Jun 11, 2009

Subsystems (3)
lore
Share & tools
lore

Michael S. Tsirkin

author maintainer
#0 lore raw

PATCH 1/1 msi-x: let drivers retry when not enough vectors

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +3 -3
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability structure
  * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
  * Or a return of > 0 indicates that driver request is exceeding the number
- * of irqs available. Driver should use the returned value to re-send
- * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to
+ * re-send its request.
  **/
 int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
 {
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */
 	for (i = 0; i < nvec; i++) {
Signature · 1 line
1.6.3.rc3.1.g830204
#1 lore raw
Quoted 6 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Signature · 2 lines
regards
Yang, Sheng
Quoted 38 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability structure
  * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
  * Or a return of > 0 indicates that driver request is exceeding the

number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int
nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

#31 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Signature · 2 lines
regards
Yang, Sheng
Quoted 38 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability structure
  * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
  * Or a return of > 0 indicates that driver request is exceeding the

number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int
nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

author maintainer
#2 lore raw
In reply to Sheng Yang
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Drivers can do this, but it's more code. Since pci_enable_msix calls pci_msix_table_size already, let it do the work. Right?

Quoted 43 lines
Signature · 2 lines
regards
Yang, Sheng
Quoted 38 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability structure
  * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
  * Or a return of > 0 indicates that driver request is exceeding the

number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int
nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

Signature · 1 line
MST
author maintainer
#3 lore raw
In reply to Sheng Yang
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Drivers can do this, but it's more code. Since pci_enable_msix calls pci_msix_table_size already, let it do the work. Right?

Quoted 43 lines
Signature · 2 lines
regards
Yang, Sheng
Quoted 38 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability structure
  * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
  * Or a return of > 0 indicates that driver request is exceeding the

number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int
nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

Signature · 1 line
MST
#4 lore raw
Quoted 9 lines · On Thursday 07 May 2009 17:05:06 Michael S. Tsirkin wrote:
Quoted 8 lines · Sheng Yang
Quoted 6 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Quoted 8 lines
Quoted 5 lines

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Drivers can do this, but it's more code. Since pci_enable_msix calls pci_msix_table_size already, let it do the work. Right?

If you don't know the vectors number before you enable MSI-X, how can you setup vectors?

I don't think it's proper way to go.

Signature · 2 lines
regards
Yang, Sheng
Quoted 42 lines
Quoted 42 lines

--
regards
Yang, Sheng

Quoted 37 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability

structure * with new allocated MSI-X irqs. A return of < 0 indicates a failure. * Or a return of > 0 indicates that driver request is exceeding the number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries,
int nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

#6 lore raw
In reply to Michael S. Tsirkin
Quoted 9 lines · On Thursday 07 May 2009 17:05:06 Michael S. Tsirkin wrote:
Quoted 8 lines · Sheng Yang
Quoted 6 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Quoted 8 lines
Quoted 5 lines

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Drivers can do this, but it's more code. Since pci_enable_msix calls pci_msix_table_size already, let it do the work. Right?

If you don't know the vectors number before you enable MSI-X, how can you setup vectors?

I don't think it's proper way to go.

Signature · 2 lines
regards
Yang, Sheng
Quoted 42 lines
Quoted 42 lines

--
regards
Yang, Sheng

Quoted 37 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

 drivers/pci/msi.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/msi.c +0 -0
index 6f2e629..f5bd1c9 100644
@@ -687,8 +687,8 @@ int pci_msix_table_size(struct pci_dev *dev)
  * indicates the successful configuration of MSI-X capability

structure * with new allocated MSI-X irqs. A return of < 0 indicates a failure. * Or a return of > 0 indicates that driver request is exceeding the number - * of irqs available. Driver should use the returned value to re-send - * its request.
+ * of irqs or MSI-X vectors available. Driver should use the returned value to + * re-send its request.
**/
int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries,
int nvec) {

drivers/pci/msi.c +1 -1
@@ -704,7 +704,7 @@ int pci_enable_msix(struct pci_dev* dev, struct
 msix_entry *entries, int nvec)
 
 	nr_entries = pci_msix_table_size(dev);
 	if (nvec > nr_entries)
-		return -EINVAL;
+		return nr_entries;
 
 	/* Check for any invalid entries */

for (i = 0; i < nvec; i++) {

Reviewed-by
#7 lore raw
In reply to Sheng Yang
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

I think Michael's patch makes sense. It reduces the amount of work the driver has to do without requiring any additional work in the core. I don't see the disadvantage to it.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
Reviewed-by
#8 lore raw
In reply to Sheng Yang
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

I think Michael's patch makes sense. It reduces the amount of work the driver has to do without requiring any additional work in the core. I don't see the disadvantage to it.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
author maintainer
#5 lore raw
In reply to Sheng Yang
Quoted 9 lines · Sheng Yang
Quoted 6 lines
Quoted 3 lines

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Drivers can do this, but it's more code. Since pci_enable_msix calls pci_msix_table_size already, let it do the work. Right?

If you don't know the vectors number before you enable MSI-X, how can you setup vectors?

I don't know how many irqs I will be able to get anyway. vectors that can't be assigned are useless ...

Signature · 1 line
MST
#9 lore raw
In reply to Matthew Wilcox
Quoted 15 lines · On Thursday 07 May 2009 17:27:31 Matthew Wilcox wrote:
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Quoted 5 lines

I think Michael's patch makes sense. It reduces the amount of work the driver has to do without requiring any additional work in the core. I don't see the disadvantage to it.

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Signature · 2 lines
regards
Yang, Sheng
#30 lore raw
In reply to Matthew Wilcox
Quoted 15 lines · On Thursday 07 May 2009 17:27:31 Matthew Wilcox wrote:
Quoted 5 lines · Sheng Yang
Quoted 4 lines · On Thursday 07 May 2009 16:28:41 Michael S. Tsirkin wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

Quoted 8 lines
Quoted 2 lines

It's better to return the table size, making the driver retry MSI-X with less vectors.

Hi Michael

I think driver should read from capability list to know how many vector supported by this device before enable MSI-X for device, as pci_msix_table_size() did...

Quoted 5 lines

I think Michael's patch makes sense. It reduces the amount of work the driver has to do without requiring any additional work in the core. I don't see the disadvantage to it.

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Signature · 2 lines
regards
Yang, Sheng
#10 lore raw
In reply to Sheng Yang
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#29 lore raw
In reply to Sheng Yang
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#11 lore raw
In reply to Matthew Wilcox
Quoted 12 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Yeah, partly understand now.

But the confusing of return value is not that pleasure compared to this benefit. And even you have to fall back if return > 0 anyway, but in the past, you just need fall back once at most; but now you may fall back twice. This make thing more complex - you need either two ifs or a simple loop. And just one "if" can deal with it before. All that required is one call for pci_msix_table_size(), and I believe most driver would like to know how much vector it have before it fill the vectors, so mostly no extra cost. But for this ambiguous return meaning, you have to add more code for fall back - yes, the driver may can assert that the positive return value always would be irq numbers if it call pci_msix_table_size() before, but is it safe in logic?

Signature · 2 lines
regards
Yang, Sheng
#20 lore raw
In reply to Matthew Wilcox
Quoted 12 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Yeah, partly understand now.

But the confusing of return value is not that pleasure compared to this benefit. And even you have to fall back if return > 0 anyway, but in the past, you just need fall back once at most; but now you may fall back twice. This make thing more complex - you need either two ifs or a simple loop. And just one "if" can deal with it before. All that required is one call for pci_msix_table_size(), and I believe most driver would like to know how much vector it have before it fill the vectors, so mostly no extra cost. But for this ambiguous return meaning, you have to add more code for fall back - yes, the driver may can assert that the positive return value always would be irq numbers if it call pci_msix_table_size() before, but is it safe in logic?

Signature · 2 lines
regards
Yang, Sheng
#21 lore raw
In reply to Matthew Wilcox
Quoted 12 lines · Matthew Wilcox
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207
208 return -ENOSPC;
209 }

So I agree, this patch is an improvement.

cheers

#28 lore raw
In reply to Matthew Wilcox
Quoted 12 lines · Matthew Wilcox
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207
208 return -ENOSPC;
209 }

So I agree, this patch is an improvement.

cheers

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization

#22 lore raw
In reply to Michael Ellerman
Quoted 36 lines · On Thursday 07 May 2009 18:23:50 Michael Ellerman wrote:
Quoted 12 lines · Matthew Wilcox
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207
208 return -ENOSPC;
209 }

Quoted 2 lines

So I agree, this patch is an improvement.

Oh yeah.

Forgot irq counts can also be changed from time to time.

OK, there should be a loop, so that's fine. :)

Signature · 2 lines
regards
Yang, Sheng
#23 lore raw
In reply to Michael Ellerman
Quoted 36 lines · On Thursday 07 May 2009 18:23:50 Michael Ellerman wrote:
Quoted 12 lines · Matthew Wilcox
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207
208 return -ENOSPC;
209 }

Quoted 2 lines

So I agree, this patch is an improvement.

Oh yeah.

Forgot irq counts can also be changed from time to time.

OK, there should be a loop, so that's fine. :)

Signature · 2 lines
regards
Yang, Sheng
author maintainer
#12 lore raw
In reply to Sheng Yang
Quoted 18 lines · Sheng Yang
Quoted 12 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:
Quoted 4 lines · Sheng Yang

It's indeed weird. Why the semantic of pci_enable_msix can be changed to "enable msix, or tell me how many vector do you have"? You can simply call pci_msix_table_size() to get what you want, also without any more work, no? I can't understand...

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Quoted 4 lines

This is less work in the driver (since it must implement falling back to a smaller number of interrupts *anyway*) than interrogating the card to find out how many interrupts there are, then requesting the right number, and still having the fallback path which is going to be less tested.

Quoted 5 lines

Yeah, partly understand now.

But the confusing of return value is not that pleasure compared to this benefit. And even you have to fall back if return > 0 anyway, but in the past, you just need fall back once at most; but now you may fall back twice.

I don't think that's right - you might not be able to get the number of interrupts that pci_enable_msix reported.

Quoted 8 lines

This
make thing more complex - you need either two ifs or a simple loop. And just one "if" can deal with it before. All that required is one call for pci_msix_table_size(), and I believe most driver would like to know how much vector it have before it fill the vectors, so mostly no extra cost. But for this ambiguous return meaning, you have to add more code for fall back - yes, the driver may can assert that the positive return value always would be irq numbers if it call pci_msix_table_size() before, but is it safe in logic?

If you know how many vectors the card has, then the only failure mode is when you are out of irqs. No change there.

Signature · 1 line
MST
#24 lore raw
In reply to Michael Ellerman
Quoted 17 lines · Michael Ellerman

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207 208 return -ENOSPC;
209 }

Quoted 2 lines

So I agree, this patch is an improvement.

I imagine this loop is present in many drivers. So why not add a helper

static int pci_enable_msix_min(struct foo_adapter *adapter, int min_nvec)

{
	int nvec = 2048;

       while (nvec >= min_nvec) {
               rc = pci_enable_msix(adapter->pdev,
                                    adapter->msix_entries, nvec);
               if (rc == 0)
                       return nvec;
               else if (rc > 0)
                       nvec = rc;
               else
                       return rc;
       }
       return -ENOSPC;
}
Signature · 1 line
error compiling committee.c: too many arguments to function
#27 lore raw
In reply to Michael Ellerman
Quoted 18 lines · Michael Ellerman

Not to mention that there's no guarantee that you'll get as many interrupts as the device supports, so you should really be coding to cope with that anyway. Like the example in MSI-HOWTO.txt:

197 static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) 198 {
199 while (nvec >= FOO_DRIVER_MINIMUM_NVEC) {
200 rc = pci_enable_msix(adapter->pdev,
201 adapter->msix_entries, nvec);
202 if (rc > 0)
203 nvec = rc;
204 else
205 return rc;
206 }
207
208 return -ENOSPC;
209 }

Quoted 2 lines

So I agree, this patch is an improvement.

I imagine this loop is present in many drivers. So why not add a helper

static int pci_enable_msix_min(struct foo_adapter *adapter, int min_nvec)

{
	int nvec = 2048;

        while (nvec >= min_nvec) {
                rc = pci_enable_msix(adapter->pdev,
                                     adapter->msix_entries, nvec);
                if (rc == 0)
                        return nvec;
                else if (rc > 0)
                        nvec = rc;
                else
                        return rc;
        }
        return -ENOSPC;
}
Signature · 1 line
error compiling committee.c: too many arguments to function
#25 lore raw
In reply to Avi Kivity
Quoted 1 line · Avi Kivity

I imagine this loop is present in many drivers. So why not add a helper

Let's look!

arch/x86/kernel/amd_iommu_init.c : Needs an exact number of vectors.
drivers/block/cciss.c : If it doesn't get all 4 vectors, falls back to pin mode (instead of MSI mode -- bug!) drivers/dma/ioat_dma.c : Falls back down to 1 vector if it can't get nvec drivers/infiniband/hw/mthca/mthca_main.c : Reverts to MSI if it can't get 3. drivers/net/benet/be_main.c : Falls back to MSI if it can't get 2.
drivers/net/bnx2.c : Falls back to MSI if it can't get 9.
drivers/net/bnx2x_main.c : Falls back to MSI if it can't get N. drivers/net/e1000e/netdev.c: Falls back to MSI if it can't get N. drivers/net/enic/enic_main.c: Falls back to MSI if it can't get N.
drivers/net/forcedeth.c: Falls back to MSI if it can't get N.
drivers/net/igb/igb_main.c: Falls back to MSI if it can't get N. drivers/net/igbvf/netdev.c: Falls back to MSI if it can't get 3. drivers/net/myri10ge/myri10ge.c: Falls back to Pin if it can't get N. drivers/net/netxen/netxen_nic_main.c: Falls back to MSI if it can't get N. drivers/net/qlge/qlge_main.c: Falls back to MSI if it can't get N.
drivers/net/s2io.c: Falls back to MSI if it can't get N.
drivers/net/vxge/vxge-main.c: Falls back once to a second number. drivers/pci/pcie/portdrv_core.c: Falls back to MSI if it can't get all of them. drivers/scsi/lpfc/lpfc_init.c: Falls back to MSI if it can't get N. drivers/scsi/mpt2sas/mpt2sas_base.c: Only allocates 1.

drivers/net/cxgb3/cxgb3_main.c: Actually falls back ... in a bit of a weird way. This one could definitely do with the proposed loop. drivers/net/ixgbe/ixgbe_main.c: Could also be improved with this loop.
drivers/net/mlx4/main.c: Nasty goto-based loop.
drivers/net/niu.c: Ditto
drivers/net/sfc/efx.c: Only falls back once. Would benefit from loop.
drivers/scsi/qla2xxx/qla_isr.c: Only falls back once. Would benefit from loop.
drivers/staging/sxg/sxg.c: /*Should try with less vector returned.*/

so ... 7 drivers would benefit from this loop. 20 seem like they wouldn't.

What a lot of drivers seem to do is fall back either to a single MSI or just pin-based interrupts when they can't get as many interrupts as they would like. They don't try to get a single MSI-X interrupt. I feel an update to the MSI-HOWTO coming on.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#26 lore raw
In reply to Avi Kivity
Quoted 1 line · Avi Kivity

I imagine this loop is present in many drivers. So why not add a helper

Let's look!

arch/x86/kernel/amd_iommu_init.c : Needs an exact number of vectors.
drivers/block/cciss.c : If it doesn't get all 4 vectors, falls back to pin mode (instead of MSI mode -- bug!) drivers/dma/ioat_dma.c : Falls back down to 1 vector if it can't get nvec drivers/infiniband/hw/mthca/mthca_main.c : Reverts to MSI if it can't get 3. drivers/net/benet/be_main.c : Falls back to MSI if it can't get 2.
drivers/net/bnx2.c : Falls back to MSI if it can't get 9.
drivers/net/bnx2x_main.c : Falls back to MSI if it can't get N. drivers/net/e1000e/netdev.c: Falls back to MSI if it can't get N. drivers/net/enic/enic_main.c: Falls back to MSI if it can't get N.
drivers/net/forcedeth.c: Falls back to MSI if it can't get N.
drivers/net/igb/igb_main.c: Falls back to MSI if it can't get N. drivers/net/igbvf/netdev.c: Falls back to MSI if it can't get 3. drivers/net/myri10ge/myri10ge.c: Falls back to Pin if it can't get N. drivers/net/netxen/netxen_nic_main.c: Falls back to MSI if it can't get N. drivers/net/qlge/qlge_main.c: Falls back to MSI if it can't get N.
drivers/net/s2io.c: Falls back to MSI if it can't get N.
drivers/net/vxge/vxge-main.c: Falls back once to a second number. drivers/pci/pcie/portdrv_core.c: Falls back to MSI if it can't get all of them. drivers/scsi/lpfc/lpfc_init.c: Falls back to MSI if it can't get N. drivers/scsi/mpt2sas/mpt2sas_base.c: Only allocates 1.

drivers/net/cxgb3/cxgb3_main.c: Actually falls back ... in a bit of a weird way. This one could definitely do with the proposed loop. drivers/net/ixgbe/ixgbe_main.c: Could also be improved with this loop.
drivers/net/mlx4/main.c: Nasty goto-based loop.
drivers/net/niu.c: Ditto
drivers/net/sfc/efx.c: Only falls back once. Would benefit from loop.
drivers/scsi/qla2xxx/qla_isr.c: Only falls back once. Would benefit from loop.
drivers/staging/sxg/sxg.c: /*Should try with less vector returned.*/

so ... 7 drivers would benefit from this loop. 20 seem like they wouldn't.

What a lot of drivers seem to do is fall back either to a single MSI or just pin-based interrupts when they can't get as many interrupts as they would like. They don't try to get a single MSI-X interrupt. I feel an update to the MSI-HOWTO coming on.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#13 lore raw
In reply to Sheng Yang
Quoted 6 lines · Sheng Yang
Quoted 5 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Sheng is absolutely right, that's a horrid API.

If it actually enabled that number and returned it, it might make sense (cf. write() returning less bytes than you give it). But overloading the return value to save an explicit call is just ugly; it's not worth saving a few lines of code at cost of making all the drivers subtle and tricksy.

Fail with -ENOSPC or something.

Rusty.

#19 lore raw
In reply to Sheng Yang
Quoted 6 lines · Sheng Yang
Quoted 5 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Sheng is absolutely right, that's a horrid API.

If it actually enabled that number and returned it, it might make sense (cf. write() returning less bytes than you give it). But overloading the return value to save an explicit call is just ugly; it's not worth saving a few lines of code at cost of making all the drivers subtle and tricksy.

Fail with -ENOSPC or something.

Rusty.

#14 lore raw
In reply to Rusty Russell
Quoted 1 line · Rusty Russell

Sheng is absolutely right, that's a horrid API.

But the API already exists, and is in use by 27 drivers. If this were a change to the API, I'd be against it, but it is the existing API.

#15 lore raw
In reply to Rusty Russell
Quoted 1 line · Rusty Russell

Sheng is absolutely right, that's a horrid API.

But the API already exists, and is in use by 27 drivers. If this were a change to the API, I'd be against it, but it is the existing API.

#16 lore raw
In reply to Rusty Russell
Quoted 9 lines · Rusty Russell
Quoted 6 lines · Sheng Yang
Quoted 5 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Sheng is absolutely right, that's a horrid API.

It's not horrid, though it is tricky - but only for drivers that care, you can still do:

if (pci_enable_msix(..))

	bail;
Quoted 2 lines

If it actually enabled that number and returned it, it might make sense (cf. write() returning less bytes than you give it).

It could do that, but I think that would be worse. The driver, on finding out it can only get a certain number of MSIs might need to make a decision about how it allocates those - eg. in a network driver, sharing them between TX/RX/management.

And in practice most of the drivers just bail if they can't get what they asked for, so enabling less than they wanted would just mean they have to go and disable them.

Quoted 3 lines

But overloading the return
value to save an explicit call is just ugly; it's not worth saving a few lines of code at cost of making all the drivers subtle and tricksy.

Looking at just this patch, I would agree, but unfortunately it's not that simple. The first limitation on the number of MSIs the driver can have is the number the device supports, that's what this patch does. But there are others, and they come out of the arch code, or even the firmware. So to implement pci_how_many_msis(), we'd need a parallel API all the way down to the arch code, or a flag to all the existing routines saying "don't really allocate, just find out". That would be horrid IMHO.

cheers

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization

author maintainer
#17 lore raw
In reply to Rusty Russell
Quoted 10 lines · Rusty Russell
Quoted 6 lines · Sheng Yang
Quoted 5 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Sheng is absolutely right, that's a horrid API.

Quoted 8 lines

If it actually enabled that number and returned it, it might make sense (cf. write() returning less bytes than you give it). But overloading the return value to save an explicit call is just ugly; it's not worth saving a few lines of code at cost of making all the drivers subtle and tricksy.

Fail with -ENOSPC or something.

Rusty.

I do agree that returning a positive value from pci_enable_msix it an ugly API (but note that this is the API that linux currently has).

Here's a wrapper that I ended up with in my driver:

static int enable_msix(struct pci_dev *dev, struct msix_entry *entries,
int *options, int noptions)

{
        int i;
        for (i = 0; i < noptions; ++i)
                if (!pci_enable_msix(dev, entries, options[i]))
                        return options[i];
        return -EBUSY;
}

This gets an array of options for # of vectors and tries them one after the other until an option that the system can support is found. On success, we get the # of vectors actually enabled, and driver can then use them as it sees fit.

Is there interest in moving something like this to pci.h?

Signature · 1 line
MST
author maintainer
#18 lore raw
In reply to Rusty Russell
Quoted 10 lines · Rusty Russell
Quoted 6 lines · Sheng Yang
Quoted 5 lines · On Thursday 07 May 2009 17:53:02 Matthew Wilcox wrote:

Here's a good example. Let's suppose you have a driver which supports two different models of cards, one has 16 MSI-X interrupts, the other has 10. You can call pci_enable_msix() asking for 16 vectors. If your card is model A, you get 16 interrupts. If your card is model B, it says "you can have 10".

Sheng is absolutely right, that's a horrid API.

Quoted 8 lines

If it actually enabled that number and returned it, it might make sense (cf. write() returning less bytes than you give it). But overloading the return value to save an explicit call is just ugly; it's not worth saving a few lines of code at cost of making all the drivers subtle and tricksy.

Fail with -ENOSPC or something.

Rusty.

I do agree that returning a positive value from pci_enable_msix it an ugly API (but note that this is the API that linux currently has).

Here's a wrapper that I ended up with in my driver:

static int enable_msix(struct pci_dev *dev, struct msix_entry *entries,
int *options, int noptions)

{
        int i;
        for (i = 0; i < noptions; ++i)
                if (!pci_enable_msix(dev, entries, options[i]))
                        return options[i];
        return -EBUSY;
}

This gets an array of options for # of vectors and tries them one after the other until an option that the system can support is found. On success, we get the # of vectors actually enabled, and driver can then use them as it sees fit.

Is there interest in moving something like this to pci.h?

Signature · 1 line
MST
Applied
#32 lore raw
In reply to Michael S. Tsirkin
Quoted 7 lines · On Thu, 7 May 2009 11:28:41 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Quoted 7 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

Applied this one to my linux-next branch; hopefully Rusty won't mind too much. :)

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Applied
#33 lore raw
In reply to Michael S. Tsirkin
Quoted 7 lines · On Thu, 7 May 2009 11:28:41 +0300 "Michael S. Tsirkin" <[email protected]> wrote:

pci_enable_msix currently returns -EINVAL if you ask for more vectors than supported by the device, which would typically cause fallback to regular interrupts.

It's better to return the table size, making the driver retry MSI-X with less vectors.

Quoted 7 lines

---

Hi Jesse,
This came up when I was adding MSI-X support to virtio pci driver, which does not know the exact table size upfront.
Could you consider this patch for 2.6.31 please?

Applied this one to my linux-next branch; hopefully Rusty won't mind too much. :)

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center