I/O space boot parameter
Subsystems (2)
PATCH 1/1 I/O space boot parameter
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
---
Documentation/kernel-parameters.txt +1 -0
@@ -1259,6 +1259,7 @@ This sorting is done to get a device order compatible with older (<= 2.4) kernels. nobfsort Don't sort PCI devices into breadth-first order. + noiospace Do not allocate I/O space unless the BIOS has done so. pcmv= [HW,PCMCIA] BadgePAD 4
drivers/pci/pci.c +3 -0
@@ -20,6 +20,7 @@#include "pci.h" unsigned int pci_pm_d3_delay =10 ; +unsigned int noiospace =0 ;/** * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
@@ -1168,6 +1169,8 @@if (*str && (str =pcibios_setup (str)) && *str) {if (!strcmp (str,"nomsi" )) {pci_no_msi (); + }else if (!strcmp (str,"noiospace" )) { + noiospace =1 ; }else {printk (KERN_ERR "PCI: Unknown option `%s'\n" , str);
drivers/pci/pci.h +1 -0
@@ -49,6 +49,7 @@#define pci_msi_quirk 0#endif extern unsigned int pci_pm_d3_delay; +extern unsigned int noiospace;#ifdef CONFIG_PCI_MSI void disable_msi_mode (struct pci_dev *dev,int pos,int type);void pci_no_msi (void );
drivers/pci/probe.c +6 -0
@@ -178,6 +178,12 @@ } res->end = res->start + (unsigned long ) sz; res->flags |=pci_calc_resource_flags (l); + +if (noiospace && !res->start && (res->flags &IORESOURCE_IO )) { + res->flags =0 ; + res->end =0 ; + } +if ((l & (PCI_BASE_ADDRESS_SPACE |PCI_BASE_ADDRESS_MEM_TYPE_MASK )) == (PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 )) {u32 szhi, lhi;
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
How prelevant are machines like this? And why are the BARs on these devices wrong?
Quoted 18 lines
Signed-off-by Dan Yeisley <[email protected]>---
diff -Naur
linux-2.6.20-org/Documentation/kernel-parameters.txtlinux-2.6.20-new/Documentation/kernel-parameters.txtDocumentation/kernel-parameters.txt +1
-0@@ -1259,6 +1259,7 @@ This sorting is done to get a device order compatible with older (<= 2.4) kernels. nobfsort Don't sort PCI devices into breadth-first order. + noiospace Do not allocate I/O space unless the BIOS has done so. pcmv= [HW,PCMCIA] BadgePAD 4diff -Naur
linux-2.6.20-org/drivers/pci/pci.clinux-2.6.20-new/drivers/pci/pci.c
drivers/pci/pci.c line 23
Quoted 4 lines
drivers/pci/pci.c +1
-0@@ -20,6 +20,7 @@#include "pci.h" unsigned int pci_pm_d3_delay =10 ; +unsigned int noiospace =0 ;
pci_no_iospace perhaps? "noiospace" isn't the best named global variable...
thanks,
greg k-h
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 3 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
I don't have any sales numbers, but I can tell you that our current systems can have up to 64 PCI buses.
I've been working with Emulex cards, and my understanding is that the BARs on the devices aren't wrong, but we can't allocate 4k of I/O space for each one. So we maintain a list in the BIOS of devices that don't actually need I/O space and then don't assign it. I've tested an a x86_64 system with 20+ adapters and saw all the disks attached without any problems.
I've changed the patch with the suggested variable name change.
---
Documentation/kernel-parameters.txt +1 -0
@@ -1259,6 +1259,7 @@ This sorting is done to get a device order compatible with older (<= 2.4) kernels. nobfsort Don't sort PCI devices into breadth-first order. + noiospace Do not allocate I/O space unless the BIOS has done so. pcmv= [HW,PCMCIA] BadgePAD 4
drivers/pci/pci.c +3 -0
@@ -20,6 +20,7 @@#include "pci.h" unsigned int pci_pm_d3_delay =10 ; +unsigned int pci_no_iospace =0 ;/** * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
@@ -1168,6 +1169,8 @@if (*str && (str =pcibios_setup (str)) && *str) {if (!strcmp (str,"nomsi" )) {pci_no_msi (); + }else if (!strcmp (str,"noiospace" )) { + pci_no_iospace =1 ; }else {printk (KERN_ERR "PCI: Unknown option `%s'\n" , str);
drivers/pci/pci.h +1 -0
@@ -49,6 +49,7 @@#define pci_msi_quirk 0#endif extern unsigned int pci_pm_d3_delay; +extern unsigned int pci_no_iospace;#ifdef CONFIG_PCI_MSI void disable_msi_mode (struct pci_dev *dev,int pos,int type);void pci_no_msi (void );
drivers/pci/probe.c +6 -0
@@ -178,6 +178,12 @@ } res->end = res->start + (unsigned long ) sz; res->flags |=pci_calc_resource_flags (l); + +if (pci_no_iospace && !res->start && (res->flags &IORESOURCE_IO )) { + res->flags =0 ; + res->end =0 ; + } +if ((l & (PCI_BASE_ADDRESS_SPACE |PCI_BASE_ADDRESS_MEM_TYPE_MASK )) == (PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 )) {u32 szhi, lhi;
Quoted 17 lines · Daniel Yeisley
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 3 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
I don't have any sales numbers, but I can tell you that our current systems can have up to 64 PCI buses.
Quoted 6 lines
I've been working with Emulex cards, and my understanding is that the BARs on the devices aren't wrong, but we can't allocate 4k of I/O space for each one. So we maintain a list in the BIOS of devices that don't actually need I/O space and then don't assign it. I've tested an a
x86_64system with 20+ adapters and saw all the disks attached without any problems.
Ah. Others are working on providing a fix for this too, but it is being done in the drivers themselves, not in the pci core. Look in the linux-pci mailing list archives for those patches (I don't think they every went into mainline for some reason, but I might be wrong...)
I suggest you work with those developers, as they have the same issue that you are trying to solve here.
thanks,
greg k-h
Quoted 25 lines · Greg KH
Quoted 17 lines · Daniel Yeisley
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 3 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
I don't have any sales numbers, but I can tell you that our current systems can have up to 64 PCI buses.
Quoted 6 lines
I've been working with Emulex cards, and my understanding is that the BARs on the devices aren't wrong, but we can't allocate 4k of I/O space for each one. So we maintain a list in the BIOS of devices that don't actually need I/O space and then don't assign it. I've tested an a
x86_64system with 20+ adapters and saw all the disks attached without any problems.
Quoted 8 lines
Ah. Others are working on providing a fix for this too, but it is being done in the drivers themselves, not in the pci core. Look in the linux-pci mailing list archives for those patches (I don't think they every went into mainline for some reason, but I might be wrong...)
I suggest you work with those developers, as they have the same issue that you are trying to solve here.
I have seen some patches that make the drivers I/O port free here:
http://lkml.org/lkml/2006/2/26/261
I checked and they still aren't in the mainline.
I don't know that it matters though because I see all the disks attached to the system regardless of whether or not the adapters get I/O space. The real issue I have is with all the error messages I get at boot. I see 40+ messages that say "PCI: Failed to allocate I/O resource..." (from setup-res.c) when the kernel tries to allocate the I/O space and can't. The modules load fine. I see all the disks just fine. But that many error messages tends to raise concerns and causes support calls from customers.
Dan
Quoted 31 lines · Daniel Yeisley
Quoted 30 lines · Greg KH
Quoted 17 lines · Daniel Yeisley
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 3 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
I don't have any sales numbers, but I can tell you that our current systems can have up to 64 PCI buses.
Quoted 6 lines
I've been working with Emulex cards, and my understanding is that the BARs on the devices aren't wrong, but we can't allocate 4k of I/O space for each one. So we maintain a list in the BIOS of devices that don't actually need I/O space and then don't assign it. I've tested an a
x86_64system with 20+ adapters and saw all the disks attached without any problems.Ah. Others are working on providing a fix for this too, but it is being done in the drivers themselves, not in the pci core. Look in the linux-pci mailing list archives for those patches (I don't think they every went into mainline for some reason, but I might be wrong...)
Quoted 6 lines
Quoted 3 lines
I suggest you work with those developers, as they have the same issue that you are trying to solve here.
I have seen some patches that make the drivers I/O port free here:
http://lkml.org/lkml/2006/2/26/261
Ah, yes, those are the ones.
Quoted 1 line
I checked and they still aren't in the mainline.
Poke the developer to get them there :)
Quoted 8 lines
I don't know that it matters though because I see all the disks attached to the system regardless of whether or not the adapters get I/O space. The real issue I have is with all the error messages I get at boot. I see 40+ messages that say "PCI: Failed to allocate I/O resource..." (from
setup-res.c) when the kernel tries to allocate the I/O space and can't. The modules load fine. I see all the disks just fine. But that many error messages tends to raise concerns and causes support calls from customers.
If this isn't an issue for functionality, why not fix your BIOS then?
And doesn't the above linked patch set also solve your issue with the noise in the syslog?
thanks,
greg k-h
Quoted 54 lines · Greg KH
Quoted 31 lines · Daniel Yeisley
Quoted 30 lines · Greg KH
Quoted 17 lines · Daniel Yeisley
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 3 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
I don't have any sales numbers, but I can tell you that our current systems can have up to 64 PCI buses.
Quoted 6 lines
I've been working with Emulex cards, and my understanding is that the BARs on the devices aren't wrong, but we can't allocate 4k of I/O space for each one. So we maintain a list in the BIOS of devices that don't actually need I/O space and then don't assign it. I've tested an a
x86_64system with 20+ adapters and saw all the disks attached without any problems.Ah. Others are working on providing a fix for this too, but it is being done in the drivers themselves, not in the pci core. Look in the linux-pci mailing list archives for those patches (I don't think they every went into mainline for some reason, but I might be wrong...)
Quoted 6 lines
Quoted 3 lines
I suggest you work with those developers, as they have the same issue that you are trying to solve here.
I have seen some patches that make the drivers I/O port free here:
http://lkml.org/lkml/2006/2/26/261Ah, yes, those are the ones.
Quoted 1 line
I checked and they still aren't in the mainline.
Poke the developer to get them there :)
Quoted 8 lines
I don't know that it matters though because I see all the disks attached to the system regardless of whether or not the adapters get I/O space. The real issue I have is with all the error messages I get at boot. I see 40+ messages that say "PCI: Failed to allocate I/O resource..." (from
setup-res.c) when the kernel tries to allocate the I/O space and can't. The modules load fine. I see all the disks just fine. But that many error messages tends to raise concerns and causes support calls from customers.
Quoted 1 line
If this isn't an issue for functionality, why not fix your BIOS then?
I'm not sure what there is to fix in the BIOS. It can't assign more than 64k of I/O space any more than the kernel can.
Quoted 4 lines
And doesn't the above linked patch set also solve your issue with the noise in the syslog?
I did try the patches and the modules load and don't request I/O space, although I only see 17 of 29 disks (I'll have to look into that more). I still get the "Failed to allocate I/O..." messages (long before the modules are loaded).
Quoted 3 lines
thanks,
greg k-h
Quoted 10 lines · Greg KH
Quoted 8 lines · Daniel Yeisley
It has been mentioned before that large systems with a lot of PCI buses have issues with the 64k I/O space limit. The ES7000 has a BIOS option to either assign I/O space to all adapters, or only to those that need it. A list of supported adapters that don't need it is kept in the BIOS. When this option is used, the kernel sees the BARs on the adapters and still tries to assign I/O space (until it runs out). I've written a patch to implement a boot parameter that tells the kernel not to assign I/O space if the BIOS hasn't.
Quoted 2 lines
How prelevant are machines like this? And why are the BARs on these devices wrong?
It is a machine scaling issue, and largely caused because we have only one PCI-IO space on x86. Since devices are increasingly moving to mmaped I/O it becomes less of an issue but there are still legacy bits and pieces.
The bridges should be able to correctly have a no pci io region, by setting base > than limit. Having bridges that can map pci io but don't have anything behind them is common.
The concept of having devices that have I/O bars that we should not assign I/O space to I find a little weird. I guess we can detect this case by simply looking to see if the bridge maps the address assigned to the bar.
Ideally the way to handle this case is to not that the BAR is not valid (but it could be) and not attempt to fix this until the driver tries to use the BAR.
The approach where we don't allocate a bar if the BIOS doesn't sounds like a hack, and we still need code in the kernel to detect that the BAR has an invalid value and that we can't use it.
I have seen so many different api's for mapping the region behind a bar that I'm a little fuzzy. Is my recollection correct that we have enough flexibility in the current API that we can detect an invalid address and allocate a valid one if needed?
I do know we have a semi common case that is related to this where the BIOS does not allocate a resource because it knows we don't need it, and the kernel being more generic decides to allocate it just in case. At which point the kernel reprograms the hardware to allocate the resource and then we have problems because the kernel didn't have enough knowledge to reprogram the root complex (northbridge) correctly.
So if we can delay our fixups to when we really need them the changes of linux working on a machine where peculiar things are happening are greater.
Eric