PCI legacy I/O port free driver (take 3)
Subsystems (5)
cover PCI legacy I/O port free driver (take 3)
Hi,
Here is an updated set of patches for PCI legacy I/O port free drivers which incorporates feedbacks. Summary of changes from the previous version are:
- Removed the device_flags field from struct pci_device_id, which was introduced in the previous version of patch
- Changed e1000 driver to use the driver_data field in struct pci_device_id to see if the device needs I/O port regions.
- Added proper messages instead of WARN_ON() at the error.
- Updated the Documentation/pci.txt
I'm attaching the following four patches:
[patch 1/4] Add no_ioport flag into pci_dev [patch 2/4] Update Documentation/pci.txt [patch 3/4] Make Intel e1000 driver legacy I/O port free [patch 4/4] Make Emulex lpfc driver legacy I/O port free
I'm attaching the brief description below about what the problem I'm trying to solve is.
Thanks,
Kenji Kaneshige
Brief Description
~~~~~~~~~~~~~~~~~
I encountered a problem that some PCI devices don't work on my system which have huge number of PCI devices.
It is mandatory for all PCI device drivers to enable the device by calling pci_enable_device() which enables all regions probed from the device's BARs. If pci_enable_device() failes to enable any regions probed from BARs, it returns as error. On the large servers, I/O port resource could not be assigned to all PCI devices because it is limited (64KB on Intel Architecture[1]) and it would be fragmented (I/O base register of PCI-to-PCI bridge will usually be aligned to a 4KB boundary[2]). In this case, the devices which have no I/O port resource assigned don't work because pci_enable_device() for those devices failes. This is what happened on my machine.
---
[1]: Some machines support 64KB I/O port space per PCI segment. [2]: Some P2P bridges support optional 1KB aligned I/O base.
Here, there are many PCI devices that provide both I/O port and MMIO interface, and some of those devices can be handled without using I/O port interface. The reason why such devices provide I/O port interface is for compatibility to legacy OSs. So this kind of devices should work even if enough I/O port resources are not assigned. The "PCI Local Bus Specification Revision 3.0" also mentions about this topic (Please see p.44, "IMPLEMENTATION NOTE"). On the current linux, unfortunately, this kind of devices don't work if I/O port resources are not assigned, because pci_enable_device() for those devices fails.
PATCH 1/4 PCI legacy I/O port free driver (take 3) - Add no_ioport flag into pci_dev
This patch adds the no_ioport field into struct pci_dev, which is used to tell the kernel not to touch any I/O port regions.
drivers/pci/pci.c | 39 +++++++++++++++++++++++++++++++-------- include/linux/pci.h | 1 + 2 files changed, 32 insertions(+), 8 deletions(-)
include/linux/pci.h +1 -0
@@ -152,6 +152,7 @@ unsigned int is_busmaster:1; /* device is busmaster */ unsigned int no_msi:1; /* device may not use msi */ unsigned int block_ucfg_access:1; /* userspace config space access is blocked */ + unsigned int no_ioport:1; /* device may not use ioport */u32 saved_config_space[16 ];/* config space saved at suspend time */ struct bin_attribute *rom_attr;/* attribute descriptor for sysfs ROM entry */
drivers/pci/pci.c +31 -8
@@ -495,9 +495,14 @@int pci_enable_device (struct pci_dev *dev) { -int err; +int err, i, bars = (1 <<PCI_NUM_RESOURCES ) -1 ; -if ((err =pci_enable_device_bars (dev, (1 <<PCI_NUM_RESOURCES ) -1 ))) +if (dev->no_ioport) +for (i =0 ; i <PCI_NUM_RESOURCES ; i++) +if (pci_resource_flags (dev, i) &IORESOURCE_IO ) + bars &= ~(1 << i); + +if ((err =pci_enable_device_bars (dev, bars)))return err;pci_fixup_device (pci_fixup_enable, dev); dev->is_enabled =1 ;
@@ -617,9 +622,14 @@ {if (pci_resource_len (pdev, bar) ==0 )return ; -if (pci_resource_flags (pdev, bar) &IORESOURCE_IO ) +if (pci_resource_flags (pdev, bar) &IORESOURCE_IO ) { +if (pdev->no_ioport) +dev_warn (&pdev->dev, +"Trying to release PCI I/O region #%d for " +"the device marked I/O resource free\n" , bar);release_region (pci_resource_start (pdev, bar),pci_resource_len (pdev, bar)); + }else if (pci_resource_flags (pdev, bar) &IORESOURCE_MEM )release_mem_region (pci_resource_start (pdev, bar),pci_resource_len (pdev, bar));
@@ -645,6 +655,10 @@return 0 ;if (pci_resource_flags (pdev, bar) &IORESOURCE_IO ) { +if (pdev->no_ioport) +dev_warn (&pdev->dev, +"Trying to request PCI I/O region #%d for " +"the device marked I/O resource free\n" , bar);if (!request_region (pci_resource_start (pdev, bar),pci_resource_len (pdev, bar), res_name)) goto err_out;
@@ -678,10 +692,13 @@void pci_release_regions (struct pci_dev *pdev) { -int i; +int i, no_ioport = pdev->no_ioport; -for (i =0 ; i <6 ; i++) +for (i =0 ; i <6 ; i++) { +if (no_ioport && (pci_resource_flags (pdev, i) &IORESOURCE_IO )) +continue ;pci_release_region (pdev, i); + } }/**
@@ -699,16 +716,22 @@*/ int pci_request_regions (struct pci_dev *pdev,char *res_name) { -int i; +int i, no_ioport = pdev->no_ioport; -for (i =0 ; i <6 ; i++) +for (i =0 ; i <6 ; i++) { +if (no_ioport && (pci_resource_flags (pdev, i) &IORESOURCE_IO )) +continue ;if (pci_request_region (pdev, i, res_name)) goto err_out; + }return 0 ; err_out: -while (--i >=0 ) +while (--i >=0 ) { +if (no_ioport && (pci_resource_flags (pdev, i) &IORESOURCE_IO )) +continue ;pci_release_region (pdev, i); + }return -EBUSY ; }
PATCH 2/4 PCI legacy I/O port free driver (take 3) - Update Documentation/pci.txt
In reply to Kenji KaneshigeThis patch adds the description about legacy I/O port free driver into Documentation/pci.txt.
Documentation/pci.txt | 28 ++++++++++++++++++++++++++++ 1 files changed, 28 insertions(+)
Documentation/pci.txt +28 -0
@@ -269,3 +269,31 @@ pci_find_device() Superseded by pci_get_device() pci_find_subsys() Superseded by pci_get_subsys() pci_find_slot() Superseded by pci_get_slot() + + +9. Legacy I/O port free driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +On the large servers, I/O port resources could not be assigned to all +PCI devices because it is limited (64KB on Intel Architecture[1]) and +it would be fragmented (I/O base register of PCI-to-PCI bridge will +usually be aligned to a 4KB boundary[2]). On such systems, +pci_enable_device() and pci_request_regions() for those devices will +fail because those functions try to enable all the regions. However, +it is a problem for some PCI devices which provide both I/O port and +MMIO interface because some of them can be handled without using I/O +port interface. The reason why such devices provide I/O port interface +is for compatibility to legacy OSs. So this kind of devices should +work even if enough I/O port resources are not assigned. The "PCI +Local Bus Specification Revision 3.0" also mentions about this topic +(Please see p.44, "IMPLEMENTATION NOTE"). + +This problem is solved by telling the kernel if your driver needs to +use I/O port to handle the device. If your driver doesn't need any I/O +port regions to handle the device, you can tell it to the kernel by +setting the no_ioport flag in struct pci_dev. If the no_ioport flag is +set, kernel will never touch I/O port regions for the corresponding +devices. Please note that you need to set the no_ioport flag before +calling pci_enable_device() and pci_request_regions(). + +[1] Some systems support 64KB I/O port space per PCI segment. +[2] Some PCI-to-PCI bridges support optional 1KB aligned I/O base.
Re: [PATCH 2/4] PCI legacy I/O port free driver (take 3) - Update Documentation/pci.txt
Quoted 2 lines · Kenji Kaneshige
This patch adds the description about legacy I/O port free driver into
Documentation/pci.txt.
Kenji,
Since you've done all the hard work so far, my small contribution is to rewrite the pci.txt so it's easier to read.
Does this look better to you?
The new section on "Write Posting" is just a proposal (RFC). Write Posting has been discussed on most linux driver mailing lists but I didn't see it mentioned in Documentation/pci.txt (which seems like a natural place to discuss it).
cheers,
grant
9. Legacy I/O port free driver
Large servers may not be able to provide I/O port resources to all PCI devices. I/O Port space is only 64KB on Intel Architecture[1] and is likely also fragmented since the I/O base register of PCI-to-PCI bridge will usually be aligned to a 4KB boundary[2]. On such systems, pci_enable_device() and pci_request_regions() will fail when attempting to enable I/O Port regions that don't have I/O Port resources assigned.
Fortunately, many PCI devices which request I/O Port resources also provide access to the same registers via MMIO BARs. These devices can be handled without using I/O port space and the drivers typically offer a CONFIG_ option to only use MMIO regions (e.g. CONFIG_TULIP_MMIO). PCI devices typically provide I/O port interface for legacy OSs and will work when I/O port resources are not assigned. The "PCI Local Bus Specification Revision 3.0" discusses this on p.44, "IMPLEMENTATION NOTE".
If your PCI device driver doesn't need I/O port resources assigned to I/O Port BARs, set the no_ioport flag in struct pci_dev before calling pci_enable_device() and pci_request_regions(). If the no_ioport flag is set, generic PCI support will ignore I/O port regions for the corresponding devices.
[1] Some systems support 64KB I/O port space per PCI segment.
[2] Some PCI-to-PCI bridges support optional 1KB aligned I/O base.
10. MMIO Space and "Write Posting"
Converting a driver from using I/O Port space to using MMIO space often requires some additional changes. Specifically, "write posting" needs to be handled. Most drivers (e.g. tg3, acenic, sym53c8xx_2) already do. I/O Port space guarantees write transactions reach the PCI device before the CPU can continue. Writes to MMIO space allow to CPU continue before the transaction reaches the PCI device. HW weenies call this "Write Posting" because the write completion is "posted" to the CPU before the transaction has reached it's destination.
Thus, timing sensitive code should add readl() where the CPU is expected to wait before doing other work. The classic "bit banging" sequence works fine for I/O Port space:
for (i=8; --i; val >>= 1) {
outb(val & 1, ioport_reg); /* write bit */
udelay(10);
}The same sequence for MMIO space should be:
for (i=8; --i; val >>= 1) {
writeb(val & 1, mmio_reg); /* write bit */
readb(safe_mmio_reg); /* flush posted write */
udelay(10);
}It is important that "safe_mmio_reg" not have any side effects that interferes with the correct operation of the device.
Another case to watch out for is when resetting a PCI device. Use PCI Configuration space reads to flush the writel(). This will gracefully handle the PCI master abort on all platforms if the PCI device is expected to not respond to a readl(). Most x86 platforms will allow MMIO reads to master abort (aka "Soft Fail") and return garbage (e.g. ~0). But many RISC platforms will crash (aka "Hard Fail").
Re: [PATCH 2/4] PCI legacy I/O port free driver (take 3) - Update Documentation/pci.txt
Quoted 11 lines · Grant Grundler
Quoted 2 lines · Kenji Kaneshige
This patch adds the description about legacy I/O port free driver into
Documentation/pci.txt.Kenji,
Since you've done all the hard work so far, my small contribution is to rewrite the pci.txt so it's easier to read.
Does this look better to you?
Thank you very much! This looks much better to me. I'll update the patch.
As you know, it is very hard work for me to write English. I usually spend much more time for that than writing C language...
Quoted 5 lines
The new section on "Write Posting" is just a proposal (RFC). Write Posting has been discussed on most linux driver mailing lists but I didn't see it mentioned in
Documentation/pci.txt(which seems like a natural place to discuss it).
I think this new section is very important and it should be added to Documentation/pci.txt. I have been wondering that some drivers might cause spurious interrupts because IIRC, some drivers don't do "Write Posting" when they update their interrupt status register through MMIO, though I don't know about it very much.
Thanks,
Kenji Kaneshige
Quoted 71 lines
cheers,
grant9. Legacy I/O port free driver
Large servers may not be able to provide I/O port resources to all PCI devices. I/O Port space is only 64KB on Intel Architecture[1] and is likely also fragmented since the I/O base register of PCI-to-PCI bridge will usually be aligned to a 4KB boundary[2]. On such systems,
pci_enable_device()andpci_request_regions()will fail when attempting to enable I/O Port regions that don't have I/O Port resources assigned.Fortunately, many PCI devices which request I/O Port resources also provide access to the same registers via MMIO BARs. These devices can be handled without using I/O port space and the drivers typically offer a
CONFIG_option to only use MMIO regions (e.g.CONFIG_TULIP_MMIO). PCI devices typically provide I/O port interface for legacy OSs and will work when I/O port resources are not assigned. The "PCI Local Bus Specification Revision 3.0" discusses this on p.44, "IMPLEMENTATION NOTE".If your PCI device driver doesn't need I/O port resources assigned to I/O Port BARs, set the
no_ioportflag in structpci_devbefore callingpci_enable_device()andpci_request_regions(). If theno_ioportflag is set, generic PCI support will ignore I/O port regions for the corresponding devices.[1] Some systems support 64KB I/O port space per PCI segment.
[2] Some PCI-to-PCI bridges support optional 1KB aligned I/O base.10. MMIO Space and "Write Posting"
Converting a driver from using I/O Port space to using MMIO space often requires some additional changes. Specifically, "write posting" needs to be handled. Most drivers (e.g. tg3, acenic,
sym53c8xx_2) already do. I/O Port space guarantees write transactions reach the PCI device before the CPU can continue. Writes to MMIO space allow to CPU continue before the transaction reaches the PCI device. HW weenies call this "Write Posting" because the write completion is "posted" to the CPU before the transaction has reached it's destination.Thus, timing sensitive code should add
readl()where the CPU is expected to wait before doing other work. The classic "bit banging" sequence works fine for I/O Port space:for (i=8; --i; val >>= 1) { outb(val & 1, ioport_reg); /* write bit */ udelay(10); }The same sequence for MMIO space should be:
for (i=8; --i; val >>= 1) { writeb(val & 1, mmio_reg); /* write bit */ readb(safe_mmio_reg); /* flush posted write */ udelay(10); }It is important that "
safe_mmio_reg" not have any side effects that interferes with the correct operation of the device.Another case to watch out for is when resetting a PCI device. Use PCI Configuration space reads to flush the
writel(). This will gracefully handle the PCI master abort on all platforms if the PCI device is expected to not respond to areadl(). Most x86 platforms will allow MMIO reads to master abort (aka "Soft Fail") and return garbage (e.g. ~0). But many RISC platforms will crash (aka "Hard Fail").
Re: [PATCH 2/4] PCI legacy I/O port free driver (take 3) - Update Documentation/pci.txt
Here is an updated patch based on the feedback from Grant Grundler.
Thanks,
Kenji Kaneshige
This patch adds the description about legacy I/O port free driver into Documentation/pci.txt.
Documentation/pci.txt | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 68 insertions(+)
Documentation/pci.txt +0 -0
@@ -269,3 +269,71 @@
pci_find_device() Superseded by pci_get_device()pci_find_subsys() Superseded by pci_get_subsys()pci_find_slot() Superseded by pci_get_slot()
+
+
+9. Legacy I/O port free driver
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Large servers may not be able to provide I/O port resources to all PCI +devices. I/O Port space is only 64KB on Intel Architecture[1] and is +likely also fragmented since the I/O base register of PCI-to-PCI +bridge will usually be aligned to a 4KB boundary[2]. On such systems, +pci_enable_device() and pci_request_regions() will fail when +attempting to enable I/O Port regions that don't have I/O Port +resources assigned.
+
+Fortunately, many PCI devices which request I/O Port resources also +provide access to the same registers via MMIO BARs. These devices can +be handled without using I/O port space and the drivers typically +offer a CONFIG_ option to only use MMIO regions
+(e.g. CONFIG_TULIP_MMIO). PCI devices typically provide I/O port +interface for legacy OSs and will work when I/O port resources are not +assigned. The "PCI Local Bus Specification Revision 3.0" discusses +this on p.44, "IMPLEMENTATION NOTE".
+
+If your PCI device driver doesn't need I/O port resources assigned to +I/O Port BARs, set the no_ioport flag in struct pci_dev before calling +pci_enable_device() and pci_request_regions(). If the no_ioport flag +is set, generic PCI support will ignore I/O port regions for the +corresponding devices.
+
+[1] Some systems support 64KB I/O port space per PCI segment. +[2] Some PCI-to-PCI bridges support optional 1KB aligned I/O base. +
+
+10. MMIO Space and "Write Posting"
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Converting a driver from using I/O Port space to using MMIO space +often requires some additional changes. Specifically, "write posting" +needs to be handled. Most drivers (e.g. tg3, acenic, sym53c8xx_2) +already do. I/O Port space guarantees write transactions reach the PCI +device before the CPU can continue. Writes to MMIO space allow to CPU +continue before the transaction reaches the PCI device. HW weenies +call this "Write Posting" because the write completion is "posted" to +the CPU before the transaction has reached it's destination. +
+Thus, timing sensitive code should add readl() where the CPU is +expected to wait before doing other work. The classic "bit banging" +sequence works fine for I/O Port space:
+
+ for (i=8; --i; val >>= 1) {
+ outb(val & 1, ioport_reg); /* write bit */
+ udelay(10);
+ }
+
+The same sequence for MMIO space should be:
+
+ for (i=8; --i; val >>= 1) {
+ writeb(val & 1, mmio_reg); /* write bit */
+ readb(safe_mmio_reg); /* flush posted write */
+ udelay(10);
+ }
+
+It is important that "safe_mmio_reg" not have any side effects that +interferes with the correct operation of the device.
+
+Another case to watch out for is when resetting a PCI device. Use PCI +Configuration space reads to flush the writel(). This will gracefully +handle the PCI master abort on all platforms if the PCI device is +expected to not respond to a readl(). Most x86 platforms will allow +MMIO reads to master abort (aka "Soft Fail") and return garbage +(e.g. ~0). But many RISC platforms will crash (aka "Hard Fail").
Re: [PATCH 2/4] PCI legacy I/O port free driver (take 3) - Update Documentation/pci.txt
The patch seemed to be broken by my mailer. Resending it. Sorry about that.
Thanks,
Kenji Kaneshige
This patch adds the description about legacy I/O port free driver into Documentation/pci.txt.
Documentation/pci.txt | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 68 insertions(+)
Documentation/pci.txt +68 -0
@@ -269,3 +269,71 @@ pci_find_device() Superseded by pci_get_device() pci_find_subsys() Superseded by pci_get_subsys() pci_find_slot() Superseded by pci_get_slot() + + +9. Legacy I/O port free driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Large servers may not be able to provide I/O port resources to all PCI +devices. I/O Port space is only 64KB on Intel Architecture[1] and is +likely also fragmented since the I/O base register of PCI-to-PCI +bridge will usually be aligned to a 4KB boundary[2]. On such systems, +pci_enable_device() and pci_request_regions() will fail when +attempting to enable I/O Port regions that don't have I/O Port +resources assigned. + +Fortunately, many PCI devices which request I/O Port resources also +provide access to the same registers via MMIO BARs. These devices can +be handled without using I/O port space and the drivers typically +offer a CONFIG_ option to only use MMIO regions +(e.g. CONFIG_TULIP_MMIO). PCI devices typically provide I/O port +interface for legacy OSs and will work when I/O port resources are not +assigned. The "PCI Local Bus Specification Revision 3.0" discusses +this on p.44, "IMPLEMENTATION NOTE". + +If your PCI device driver doesn't need I/O port resources assigned to +I/O Port BARs, set the no_ioport flag in struct pci_dev before calling +pci_enable_device() and pci_request_regions(). If the no_ioport flag +is set, generic PCI support will ignore I/O port regions for the +corresponding devices. + +[1] Some systems support 64KB I/O port space per PCI segment. +[2] Some PCI-to-PCI bridges support optional 1KB aligned I/O base. + + +10. MMIO Space and "Write Posting" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Converting a driver from using I/O Port space to using MMIO space +often requires some additional changes. Specifically, "write posting" +needs to be handled. Most drivers (e.g. tg3, acenic, sym53c8xx_2) +already do. I/O Port space guarantees write transactions reach the PCI +device before the CPU can continue. Writes to MMIO space allow to CPU +continue before the transaction reaches the PCI device. HW weenies +call this "Write Posting" because the write completion is "posted" to +the CPU before the transaction has reached it's destination. + +Thus, timing sensitive code should add readl() where the CPU is +expected to wait before doing other work. The classic "bit banging" +sequence works fine for I/O Port space: + + for (i=8; --i; val >>= 1) { + outb(val & 1, ioport_reg); /* write bit */ + udelay(10); + } + +The same sequence for MMIO space should be: + + for (i=8; --i; val >>= 1) { + writeb(val & 1, mmio_reg); /* write bit */ + readb(safe_mmio_reg); /* flush posted write */ + udelay(10); + } + +It is important that "safe_mmio_reg" not have any side effects that +interferes with the correct operation of the device. + +Another case to watch out for is when resetting a PCI device. Use PCI +Configuration space reads to flush the writel(). This will gracefully +handle the PCI master abort on all platforms if the PCI device is +expected to not respond to a readl(). Most x86 platforms will allow +MMIO reads to master abort (aka "Soft Fail") and return garbage +(e.g. ~0). But many RISC platforms will crash (aka "Hard Fail").
PATCH 3/4 PCI legacy I/O port free driver (take 3) - Make Intel e1000 driver legacy I/O port free
In reply to Kenji KaneshigeThis patch makes Intel e1000 driver legacy I/O port free.
--- drivers/net/e1000/e1000.h | 5 + drivers/net/e1000/e1000_main.c | 109 +++++++++++++++++++++-------------------- 2 files changed, 61 insertions(+), 53 deletions(-)
drivers/net/e1000/e1000.h +3 -2
@@ -77,8 +77,9 @@#define BAR_1 1#define BAR_5 5 -#define INTEL_E1000_ETHERNET_DEVICE (device_id) {\ - PCI_DEVICE(PCI_VENDOR_ID_INTEL, device_id)} +#define E1000_NO_IOPORT (1 << 0) +#define INTEL_E1000_ETHERNET_DEVICE (device_id, flags) {\ + PCI_DEVICE(PCI_VENDOR_ID_INTEL, device_id), .driver_data = flags}struct e1000_adapter ;
drivers/net/e1000/e1000_main.c +58 -51
127 lines not on this page · raw message
PATCH 4/4 PCI legacy I/O port free driver (take 3) - Make Emulex lpfc driver legacy I/O port free
In reply to Kenji KaneshigeThis patch makes Emulex lpfc driver legacy I/O port free.
--- drivers/scsi/lpfc/lpfc_init.c | 3 +++ 1 files changed, 3 insertions(+)
drivers/scsi/lpfc/lpfc_init.c +3 -0
@@ -1368,6 +1368,9 @@int i;uint16_t iotag; +/* Don't need to use I/O port regions */ + pdev->no_ioport =1 ; +if (pci_enable_device (pdev)) goto out;if (pci_request_regions (pdev,LPFC_DRIVER_NAME ))
Quoted 35 lines · Kenji Kaneshige
Hi,
Here is an updated set of patches for PCI legacy I/O port free drivers which incorporates feedbacks. Summary of changes from the previous version are:
- Removed the
device_flagsfield from structpci_device_id, which was introduced in the previous version of patch- Changed e1000 driver to use the
driver_datafield in structpci_device_idto see if the device needs I/O port regions.- Added proper messages instead of
WARN_ON()at the error.- Updated the
Documentation/pci.txtI'm attaching the following four patches:
[patch 1/4] Add
no_ioportflag intopci_dev[patch 2/4] UpdateDocumentation/pci.txt[patch 3/4] Make Intel e1000 driver legacy I/O port free [patch 4/4] Make Emulex lpfc driver legacy I/O port freeI'm attaching the brief description below about what the problem I'm trying to solve is.
Thanks,
Kenji KaneshigeBrief Description
~~~~~~~~~~~~~~~~~
I encountered a problem that some PCI devices don't work on my system which have huge number of PCI devices.
Quoted 10 lines
It is mandatory for all PCI device drivers to enable the device by calling
pci_enable_device()which enables all regions probed from the device's BARs. Ifpci_enable_device()failes to enable any regions probed from BARs, it returns as error. On the large servers, I/O port resource could not be assigned to all PCI devices because it is limited (64KB on Intel Architecture[1]) and it would be fragmented (I/O base register of PCI-to-PCI bridge will usually be aligned to a 4KB boundary[2]). In this case, the devices which have no I/O port resource assigned don't work becausepci_enable_device()for those devices failes. This is what happened on my machine.
This series still leaves a lot to be desired, and creates unnecessary driver churn. The better solution is:
1) pci_enable_device() enables what it can
2) Drivers, as they already do, will fail if they cannot map the desired memory or IO resources that are needed.
Thus, the PCI layer needs only to do #1, and existing driver code handles the rest of the situation as one currently expects.
Jeff
Quoted 2 lines · Jeff Garzik
This series still leaves a lot to be desired, and creates unnecessary driver churn.
This is a pretty small change and is not necessary for every driver. What do you still desire that this patch set doesn't provide?
Quoted 9 lines
The better solution is:
1)
pci_enable_device()enables what it can2) Drivers, as they already do, will fail if they cannot map the desired memory or IO resources that are needed.
Thus, the PCI layer needs only to do #1, and existing driver code handles the rest of the situation as one currently expects.
If in #1 pci_enable_device() assigns I/O Port resources even though the driver doesn't need it, PCI devices which _only_ support I/O Port space will get screwed (depending on config). We are trying to avoid that. Or do you have another way of avoiding unused resource allocation?
thanks,
grant
Quoted 7 lines · Grant Grundler
Quoted 2 lines · Jeff Garzik
This series still leaves a lot to be desired, and creates unnecessary driver churn.
This is a pretty small change and is not necessary for every driver.
The latter is decidedly false. The change makes no sense at all unless you update every conceivable driver that will be used on the target platform. You will always be patching drivers as users stick new cards in the target hardware.
Quoted 7 lines
Quoted 7 lines
The better solution is:
1)
pci_enable_device()enables what it can2) Drivers, as they already do, will fail if they cannot map the desired memory or IO resources that are needed.
Quoted 8 lines
Quoted 2 lines
Thus, the PCI layer needs only to do #1, and existing driver code handles the rest of the situation as one currently expects.
If in #1
pci_enable_device()assigns I/O Port resources even though the driver doesn't need it, PCI devices which _only_ support I/O Port space will get screwed (depending on config). We are trying to avoid that. Or do you have another way of avoiding unused resource allocation?
Fix the [firmware | device load order] to allocate I/O ports first to the hardware that only supports IO port accesses. Problem solved with zero kernel mods...
Jeff
Quoted 4 lines · Jeff Garzik
Or do you have another way of avoiding unused resource allocation?
Fix the [firmware | device load order] to allocate I/O ports first to the hardware that only supports IO port accesses.
How should the firmware know what hardware needs io ports and what hardware doesn't? I don't think it will scale well to put long lists of PCI-IDs into their firmware.
The driver is really the natural place to put such knowledge.
-Andi
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Jeff Garzik
Or do you have another way of avoiding unused resource allocation?
Fix the [firmware | device load order] to allocate I/O ports first to the hardware that only supports IO port accesses.
How should the firmware know what hardware needs io ports and what hardware doesn't? I don't think it will scale well to put long lists of PCI-IDs into their firmware.
Its trivial to detect PCI hardware that doesn't support MMIO, the "IO-only" hardware Grant mentioned...
Jeff
Quoted 2 lines · Jeff Garzik
Its trivial to detect PCI hardware that doesn't support MMIO, the "IO-only" hardware Grant mentioned...
But there might be hardware that needs both PIO and MMIO e.g. graphic cards.
How would the poor firmware distingush between those and MMIO only?
-Andi
Quoted 7 lines · Jeff Garzik
Quoted 6 lines · Grant Grundler
Quoted 2 lines · Jeff Garzik
This series still leaves a lot to be desired, and creates unnecessary driver churn.
Quoted 6 lines
Quoted 1 line
This is a pretty small change and is not necessary for every driver.
The latter is decidedly false. The change makes no sense at all unless you update every conceivable driver that will be used on the target platform. You will always be patching drivers as users stick new cards in the target hardware.
It's sufficiently true if 90% of the cards plugged into the machine are covered by 3 or 4 drivers. The remaining 10% can use IO port space and everything will work fine.
Quoted 8 lines
Quoted 4 lines
If in #1
pci_enable_device()assigns I/O Port resources even though the driver doesn't need it, PCI devices which _only_ support I/O Port space will get screwed (depending on config). We are trying to avoid that. Or do you have another way of avoiding unused resource allocation?Fix the [firmware | device load order] to allocate I/O ports first to the hardware that only supports IO port accesses. Problem solved with zero kernel mods...
After we (OS developers) have been telling firmware people to NOT configure every device and run it's BIOS driver at power up?
OS "hotplug" now deals with resource assignment and device initialization in most cases on these boxes. Running the BIOS/EFI driver to configure each device just takes too long.
grant
Quoted 49 lines · Jeff Garzik
Quoted 35 lines · Kenji Kaneshige
Hi,
Here is an updated set of patches for PCI legacy I/O port free drivers which incorporates feedbacks. Summary of changes from the previous version are:
- Removed the
device_flagsfield from structpci_device_id, which was introduced in the previous version of patch- Changed e1000 driver to use the
driver_datafield in structpci_device_idto see if the device needs I/O port regions.- Added proper messages instead of
WARN_ON()at the error.- Updated the
Documentation/pci.txtI'm attaching the following four patches:
[patch 1/4] Add
no_ioportflag intopci_dev[patch 2/4] UpdateDocumentation/pci.txt
[patch 3/4] Make Intel e1000 driver legacy I/O port free [patch 4/4] Make Emulex lpfc driver legacy I/O port freeI'm attaching the brief description below about what the problem I'm trying to solve is.
Thanks,
Kenji KaneshigeBrief Description
~~~~~~~~~~~~~~~~~
I encountered a problem that some PCI devices don't work on my system which have huge number of PCI devices.Quoted 10 lines
It is mandatory for all PCI device drivers to enable the device by calling
pci_enable_device()which enables all regions probed from the device's BARs. Ifpci_enable_device()failes to enable any regions probed from BARs, it returns as error. On the large servers, I/O port resource could not be assigned to all PCI devices because it is limited (64KB on Intel Architecture[1]) and it would be fragmented (I/O base register of PCI-to-PCI bridge will usually be aligned to a 4KB boundary[2]). In this case, the devices which have no I/O port resource assigned don't work becausepci_enable_device()for those devices failes. This is what happened on my machine.
Quoted 5 lines
This series still leaves a lot to be desired, and creates unnecessary driver churn. The better solution is:
1)
pci_enable_device()enables what it can
I guess your idea is changing pci_enable_device() not to return as error even if it fails to enable some regions. Is it correct? If yes, we need to change all architecture dependent code (e.g. pcibios_enable_device()) to do that, and it would need much bigger change.
Quoted 6 lines
2) Drivers, as they already do, will fail if they cannot map the desired memory or IO resources that are needed.
Thus, the PCI layer needs only to do #1, and existing driver code handles the rest of the situation as one currently expects.
Many driver uses pci_request_regions() instead of calling pci_request_region() for each region. That is, we need to consider the same problem at pci_request_regions() time. Converting drivers to use pci_request_region() instead of pci_request_regions() requires many changes and it would be troublesome for driver writers.
Thanks,
Kenji Kaneshige