ARM: PCI: implement generic PCI host controller
Subsystems (5)
v2 cover ARM: PCI: implement generic PCI host controller
Hello again,
This is version 2 of the patches I originally posted here:
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-February/229679.html
Changes since v1 include:
- Complete rename of files, comments and compatible strings to remove references to a `virtual' host controller
- Support for ECAM (depending on matched compatible string)
- Support for multiple mem resources
- Correct use of resource offsets (I think...)
- Removed explicit bridge enabling from bios32.c after discussion with Bjorn (pending testing/feedback from rmk)
Like v1, I continue to support only a single controller and therefore a single I/O space. I'm not sure how to represent multiple controllers in the device-tree without inventing some horrible hacks, so any ideas in this area would be much appreciated.
I still need to take the step of moving parts of this into a library, but I'd like to make sure the code is correct first. Tested on TC2 running KVM with kvmtool and virtio-pci.
All feedback welcome (and thanks to Jason and Arnd for their comments on v1),
Will
Will Deacon (3):
ARM: mach-virt: allow PCI support to be selected
ARM: bios32: use pci_enable_resource to enable PCI resources
PCI: ARM: add support for generic PCI host controller
.../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ arch/arm/kernel/bios32.c | 37 +-- arch/arm/mach-virt/Kconfig | 1 + drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 6 files changed, 381 insertions(+), 34 deletions(-) create mode 100644 Documentation/devicetree/bindings/pci/arm-generic-pci.txt create mode 100644 drivers/pci/host/pci-arm-generic.c
Signature · 1 line
1.8.2.2
PATCH v2 1/3 ARM: mach-virt: allow PCI support to be selected
mach-virt can make use of virtio-pci devices, which requires the guest kernel to have PCI support selected.
This patch selects CONFIG_MIGHT_HAVE_PCI when CONFIG_ARCH_VIRT=y.
--- arch/arm/mach-virt/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm/mach-virt/Kconfig +1 -0
@@ -8,3 +8,4 @@ config ARCH_VIRT select CPU_V7 select SPARSE_IRQ select USE_OF + select MIGHT_HAVE_PCI
Signature · 1 line
1.8.2.2
PATCH v2 2/3 ARM: bios32: use pci_enable_resource to enable PCI resources
In reply to Will DeaconThis patch moves bios32 over to using the generic code for enabling PCI resources. Since the core code takes care of bridge resources too, we can also drop the explicit IO and MEMORY enabling for them in the arch code.
A side-effect of this change is that we no longer explicitly enable devices when running in PCI_PROBE_ONLY mode. This stays closer to the meaning of the option and prevents us from trying to enable devices without any assigned resources (the core code refuses to enable resources without parents).
--- arch/arm/kernel/bios32.c | 37 +++---------------------------------- 1 file changed, 3 insertions(+), 34 deletions(-)
arch/arm/kernel/bios32.c +3 -34
@@ -608,41 +608,10 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,*/ int pcibios_enable_device (struct pci_dev *dev,int mask) { -u16 cmd, old_cmd; -int idx; -struct resource *r; - -pci_read_config_word (dev,PCI_COMMAND , &cmd); - old_cmd = cmd; -for (idx =0 ; idx <6 ; idx++) { -/* Only set up the requested stuff */ -if (!(mask & (1 << idx))) -continue ; - - r = dev->resource + idx; -if (!r->start && r->end) { -printk (KERN_ERR "PCI: Device %s not available because" -" of resource collisions\n" ,pci_name (dev)); -return -EINVAL ; - } -if (r->flags &IORESOURCE_IO ) - cmd |=PCI_COMMAND_IO ; -if (r->flags &IORESOURCE_MEM ) - cmd |=PCI_COMMAND_MEMORY ; - } +if (pci_has_flag (PCI_PROBE_ONLY )) +return 0 ; -/* -* Bridges (eg, cardbus bridges) need to be fully enabled -*/ -if ((dev->class >>16 ) ==PCI_BASE_CLASS_BRIDGE ) - cmd |=PCI_COMMAND_IO |PCI_COMMAND_MEMORY ; - -if (cmd != old_cmd) { -printk ("PCI: enabling device %s (%04x -> %04x)\n" , -pci_name (dev), old_cmd, cmd); -pci_write_config_word (dev,PCI_COMMAND , cmd); - } -return 0 ; +return pci_enable_resources (dev, mask); } int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
Signature · 1 line
1.8.2.2
PATCH v2 2/3 ARM: bios32: use pci_enable_resource to enable PCI resources
Quoted 4 lines · Will Deacon
This patch moves bios32 over to using the generic code for enabling PCI resources. Since the core code takes care of bridge resources too, we can also drop the explicit IO and MEMORY enabling for them in the arch code.
PCI: bus1: Fast back to back transfers disabled pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe001ffff] pci 0000:00:01.0: PCI bridge to [bus 01]
pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe00fffff]
pci 0000:00:01.0: enabling device (0140 -> 0142)
Jason
PATCH v2 2/3 ARM: bios32: use pci_enable_resource to enable PCI resources
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Will Deacon
This patch moves bios32 over to using the generic code for enabling PCI resources. Since the core code takes care of bridge resources too, we can also drop the explicit IO and MEMORY enabling for them in the arch code.
Quoted 7 lines
PCI: bus1: Fast back to back transfers disabled
pci_bus0000:01:busn_res: [bus 01-ff] end is updated to 01 pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe001ffff] pci 0000:00:01.0: PCI bridge to [bus 01]
pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe00fffff]
pci 0000:00:01.0: enabling device (0140 -> 0142)
That's really helpful. Thanks for testing, Jason!
Will
PATCH v2 2/3 ARM: bios32: use pci_enable_resource to enable PCI resources
In reply to Will DeaconQuoted 6 lines · Will Deacon
This patch moves bios32 over to using the generic code for enabling PCI resources. Since the core code takes care of bridge resources too, we can also drop the explicit IO and MEMORY enabling for them in the arch code.
Quoted 7 lines
A side-effect of this change is that we no longer explicitly enable devices when running in
PCI_PROBE_ONLYmode. This stays closer to the meaning of the option and prevents us from trying to enable devices without any assigned resources (the core code refuses to enable resources without parents).Signed-off-by Will Deacon <[email protected]>
I tested this patch with NIC on Exynos platform.
It works properly.
[ 0.406431] PCI: bus1: Fast back to back transfers disabled [ 0.411912] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [ 0.418608] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01 [ 0.425351] pci 0000:00:00.0: BAR 8: assigned [mem 0x40100000-0x401fffff] [ 0.432144] pci 0000:00:00.0: BAR 9: assigned [mem 0x40200000-0x402fffff pref] [ 0.439426] pci 0000:00:00.0: BAR 7: assigned [io 0x1000-0x1fff] [ 0.445599] pci 0000:01:00.0: BAR 1: assigned [mem 0x40100000-0x4017ffff] [ 0.452488] pci 0000:01:00.0: BAR 6: assigned [mem 0x40200000-0x4023ffff pref] [ 0.459745] pci 0000:01:00.0: BAR 0: assigned [mem 0x40180000-0x4019ffff] [ 0.466635] pci 0000:01:00.0: BAR 3: assigned [mem 0x401a0000-0x401a3fff] [ 0.473499] pci 0000:01:00.0: BAR 2: assigned [io 0x1000-0x101f] [ 0.479656] pci 0000:00:00.0: PCI bridge to [bus 01]
Best regards,
Jingoo Han
Quoted 55 lines
--- arch/arm/kernel/bios32.c | 37 +++---------------------------------- 1 file changed, 3 insertions(+), 34 deletions(-)arch/arm/kernel/bios32.c +3
-35@@ -608,41 +608,10 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,*/ int pcibios_enable_device (struct pci_dev *dev,int mask) { -u16 cmd, old_cmd; -int idx; -struct resource *r; - -pci_read_config_word (dev,PCI_COMMAND , &cmd); - old_cmd = cmd; -for (idx =0 ; idx <6 ; idx++) { -/* Only set up the requested stuff */ -if (!(mask & (1 << idx))) -continue ; - - r = dev->resource + idx; -if (!r->start && r->end) { -printk (KERN_ERR "PCI: Device %s not available because" -" of resource collisions\n" ,pci_name (dev)); -return -EINVAL ; - } -if (r->flags &IORESOURCE_IO ) - cmd |=PCI_COMMAND_IO ; -if (r->flags &IORESOURCE_MEM ) - cmd |=PCI_COMMAND_MEMORY ; - } +if (pci_has_flag (PCI_PROBE_ONLY )) +return 0 ; -/* -* Bridges (eg, cardbus bridges) need to be fully enabled -*/ -if ((dev->class >>16 ) ==PCI_BASE_CLASS_BRIDGE ) - cmd |=PCI_COMMAND_IO |PCI_COMMAND_MEMORY ; - -if (cmd != old_cmd) { -printk ("PCI: enabling device %s (%04x -> %04x)\n" , -pci_name (dev), old_cmd, cmd); -pci_write_config_word (dev,PCI_COMMAND , cmd); - } -return 0 ; +return pci_enable_resources (dev, mask); } int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma, --1.8.2.2
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Will DeaconThis patch adds support for a generic PCI host controller, such as a firmware-initialised device with static windows or an emulation by something such as kvmtool.
The controller itself has no configuration registers and has its address spaces described entirely by the device-tree (using the bindings from ePAPR). Both CAM and ECAM are supported for Config Space accesses.
Corresponding documentation is added for the DT binding.
--- .../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 4 files changed, 377 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/arm-generic-pci.txt create mode 100644 drivers/pci/host/pci-arm-generic.c
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +51 -0
@@ -0,0 +1,51 @@ +* ARM generic PCI host controller + +Firmware-initialised PCI host controllers and PCI emulations, such as the +virtio-pci implementations found in kvmtool and other para-virtualised +systems, do not require driver support for complexities such as regulator and +clock management. In fact, the controller may not even require the +configuration of a control interface by the operating system, instead +presenting a set of fixed windows describing a subset of IO, Memory and +Configuration Spaces. + +Such a controller can be described purely in terms of the standardized device +tree bindings communicated in pci.txt: + +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively) + +- ranges : As described in IEEE Std 1275-1994, but must provide + at least a definition of one or both of IO and Memory + Space. + +- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus. + +Configuration Space is assumed to be memory-mapped (as opposed to being +accessed via an ioport) and laid out with a direct correspondence to the +geography of a PCI bus address by concatenating the various components to form +an offset. + +For CAM, this 24-bit offset is: + + cfg_offset(bus, device, function, register) = + bus << 16 | device << 11 | function << 8 | register + +Whilst ECAM extends this by 4 bits to accomodate 4k of function space: + + cfg_offset(bus, device, function, register) = + bus << 20 | device << 15 | function << 12 | register + +Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties: + +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>
drivers/pci/host/Kconfig +7 -0
@@ -33,4 +33,11 @@ config PCI_RCAR_GEN2 There are 3 internal PCI controllers available with a single built-in EHCI/OHCI host controller present on each one. +config PCI_ARM_GENERIC + bool "ARM generic PCI host controller" + depends on ARM && OF + help + Say Y here if you want to support a simple generic PCI host + controller, such as the one emulated by kvmtool. + endmenu
drivers/pci/host/Makefile +1 -0
@@ -4,3 +4,4 @@ obj-$(CONFIG_PCI_IMX6) += pci-imx6.o obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o obj-$(CONFIG_PCI_TEGRA) += pci-tegra.o obj-$(CONFIG_PCI_RCAR_GEN2) += pci-rcar-gen2.o +obj-$(CONFIG_PCI_ARM_GENERIC) += pci-arm-generic.o
drivers/pci/host/pci-arm-generic.c new file +318 -0
318 lines not on this page · raw message
Signature · 1 line
1.8.2.2
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 20
Quoted 3 lines · On Wednesday 12 February 2014 20:16:11 Will Deacon wrote:
+- ranges : As described in IEEE Std 1275-1994, but must provide
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +2
-0+ at least a definition of one or both of IO and Memory + Space.
I'd say *must* provide at least non-prefetchable memory. *may* provide prefetchable memory and/or I/O space.
Quoted 20 lines
+- #address-cells : Must be 3
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +19
-0+ +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus. + +Configuration Space is assumed to be memory-mapped (as opposed to being +accessed via an ioport) and laid out with a direct correspondence to the +geography of a PCI bus address by concatenating the various components to form +an offset. + +For CAM, this 24-bit offset is: + + cfg_offset(bus, device, function, register) = + bus << 16 | device << 11 | function << 8 | register + +Whilst ECAM extends this by 4 bits to accomodate 4k of function space: + + cfg_offset(bus, device, function, register) =
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 51
Quoted 10 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +10
-0+ bus << 20 | device << 15 | function << 12 | register + +Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties: + +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>
We probably want to add an optional 'bus-range' property (the default being <0 255> if absent), so we don't have to map all the config space.
Quoted 6 lines
drivers/pci/host/Kconfig +0
-0@@ -33,4 +33,11 @@ config PCI_RCAR_GEN2 There are 3 internal PCI controllers available with a single
drivers/pci/host/Kconfig config PCI_RCAR_GEN2 · line 43
Quoted 10 lines
drivers/pci/host/Kconfig +7
-0built-in EHCI/OHCI host controller present on each one. +config PCI_ARM_GENERIC + bool "ARM generic PCI host controller" + depends on ARM && OF + help + Say Y here if you want to support a simple generic PCI host + controller, such as the one emulated by kvmtool. + endmenu
Looks good for now. In the long run, I'd hope to get rid of the 'ARM' part here and make it work on any architecture, but that requires significant work that we should not depend on here.
drivers/pci/host/Kconfig
Quoted 13 lines
drivers/pci/host/Kconfig +13
-0+struct gen_pci_cfg_window { + u64 cpu_phys; + void __iomem *base; + u8 bus; + spinlock_t lock; + const struct gen_pci_cfg_accessors *accessors; +}; + +struct gen_pci_resource { + struct list_head list; + struct resource cpu_res; + resource_size_t offset; +};
Your gen_pci_resource is actually identical to struct pci_host_bridge_window, which I guess is coincidence, but it would be nice to actually use the standard structure to make it easier to integrate with common infrastructure later.
drivers/pci/host/Kconfig
Quoted 7 lines
drivers/pci/host/Kconfig +7
-0+ +struct gen_pci { + struct device *dev; + struct resource *io_res; + struct list_head mem_res; + struct gen_pci_cfg_window cfg; +};
How about making this structure derived from pci_host_bridge? That would already contain a lot of the members, and gets two things right:
* it's useful to have an embedded 'struct device' here, and use dev->parent to point to the device from DT
* for I/O, we actually want a pci_host_bridge_window, not just a resource, since we should keep track of the offset
Quoted 10 lines
drivers/pci/host/Kconfig +10
-0+static void __iomem *gen_pci_map_cfg_bus_ecam(struct pci_bus *bus, + unsigned int devfn, + int where) +{ + struct pci_sys_data *sys = bus->sysdata; + struct gen_pci *pci = sys->private_data; + u32 busn = bus->number; + + spin_lock(&pci->cfg.lock); + if (pci->cfg.bus != busn) {
drivers/pci/host/Kconfig
Quoted 10 lines
drivers/pci/host/Kconfig +10
-0+ resource_size_t offset; + + devm_iounmap(pci->dev, pci->cfg.base); + offset = pci->cfg.cpu_phys + (busn << 20); + pci->cfg.base = devm_ioremap(pci->dev, offset, SZ_1M); + pci->cfg.bus = busn; + } + + return pci->cfg.base + ((devfn << 12) | where); +}
Nice idea, but unfortunately broken: we need config space access from atomic context, since there are drivers doing that.
drivers/pci/host/Kconfig
Quoted 10 lines
drivers/pci/host/Kconfig +10
-0+static int gen_pci_probe(struct platform_device *pdev) +{ + struct hw_pci hw; + struct of_pci_range range; + struct of_pci_range_parser parser; + struct gen_pci *pci; + const __be32 *reg; + const struct of_device_id *of_id; + struct device *dev = &pdev->dev; + struct device_node *np = dev->of_node;
Could you try to move almost all of this function into gen_pci_setup()? I suspect this will make it easier later to share this driver with other architectures.
drivers/pci/host/Kconfig
Quoted 12 lines
drivers/pci/host/Kconfig +12
-0+ + hw = (struct hw_pci) { + .nr_controllers = 1, + .private_data = (void **)&pci, + .setup = gen_pci_setup, + .map_irq = of_irq_parse_and_map_pci, + .ops = &gen_pci_ops, + }; + + pci_common_init_dev(dev, &hw); + return 0; +}
A missing part here seems to be a way to propagate errors from the pci_common_init_dev or gen_pci_setup back to the caller.
This seems to be a result of the arm pcibios implementation not being meant for loadable modules, but I suspect it can be fixed. The nr_controllers>1 logic gets a bit in the way there, but it's also a model that seems to be getting out of fashion:
kirkwood/dove/orion5x/mv78xx0 use it at the moment, but are migrating over to the new pci-mvebu code that doesn't as they get rid of the non-DT probing. pci-rcar-gen2.c uses it, but it seems they already ran into problems with that and are changing it. That pretty much leaves iop13xx as the only user, but at that point we can probably move the loop into iop13xx specific code.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Hi Arnd,
Thanks again for the comments.
Quoted 7 lines · Arnd Bergmann
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 20
Quoted 3 lines · On Wednesday 12 February 2014 20:16:11 Will Deacon wrote:
+- ranges : As described in IEEE Std 1275-1994, but must provide
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +2
-0+ at least a definition of one or both of IO and Memory + Space.I'd say *must* provide at least non-prefetchable memory. *may* provide prefetchable memory and/or I/O space.
Can do. Should I enforce this in the driver too? (i.e. complain if non-prefetchable memory is absent).
Quoted 11 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 51
Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +8
-0+Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties: + +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>We probably want to add an optional 'bus-range' property (the default being <0 255> if absent), so we don't have to map all the config space.
Yes, and that will be important given your comments later on.
Quoted 8 lines
Quoted 8 lines
drivers/pci/host/Kconfig +0
-0@@ -33,4 +33,11 @@ config PCI_RCAR_GEN2 There are 3 internal PCI controllers available with a single built-in EHCI/OHCI host controller present on each one.
Quoted 12 lines
drivers/pci/host/Kconfig config PCI_RCAR_GEN2 · line 43
Quoted 8 lines
drivers/pci/host/Kconfig +7
-0+config PCI_ARM_GENERIC + bool "ARM generic PCI host controller" + depends on ARM && OF + help + Say Y here if you want to support a simple generic PCI host + controller, such as the one emulated by kvmtool. + endmenuLooks good for now. In the long run, I'd hope to get rid of the 'ARM' part here and make it work on any architecture, but that requires significant work that we should not depend on here.
Agreed. arm64 is the obvious next target (once Liviu's series is sorted out -- I'm currently using a simplified port of bios32.c for testing).
Quoted 10 lines
Quoted 10 lines
drivers/pci/host/Kconfig +10
-0+struct gen_pci_cfg_window { + u64 cpu_phys; + void __iomem *base; + u8 bus; + spinlock_t lock; + const struct gen_pci_cfg_accessors *accessors; +}; + +struct gen_pci_resource { + struct list_head list;
Quoted 8 lines
drivers/pci/host/Kconfig
Quoted 3 lines
drivers/pci/host/Kconfig +3
-0+ struct resource cpu_res; + resource_size_t offset; +};Your
gen_pci_resourceis actually identical to structpci_host_bridge_window, which I guess is coincidence, but it would be nice to actually use the standard structure to make it easier to integrate with common infrastructure later.
Ha, at least I managed to come up with the same struct! I'll move to the generic version.
Quoted 12 lines
drivers/pci/host/Kconfig
Quoted 7 lines
drivers/pci/host/Kconfig +7
-0+ +struct gen_pci { + struct device *dev; + struct resource *io_res; + struct list_head mem_res; + struct gen_pci_cfg_window cfg; +};How about making this structure derived from
pci_host_bridge? That would already contain a lot of the members, and gets two things right:
Quoted 4 lines
* it's useful to have an embedded 'struct device' here, and use
dev->parentto point to the device from DT
* for I/O, we actually want apci_host_bridge_window, not just a resource, since we should keep track of the offset
Sure. Also, if we kill nr_controllers, then we can have a simple I/O space allocator to populate the offset.
Quoted 15 lines
Quoted 15 lines
drivers/pci/host/Kconfig +15
-0+static void __iomem *gen_pci_map_cfg_bus_ecam(struct pci_bus *bus, + unsigned int devfn, + int where) +{ + struct pci_sys_data *sys = bus->sysdata; + struct gen_pci *pci = sys->private_data; + u32 busn = bus->number; + + spin_lock(&pci->cfg.lock); + if (pci->cfg.bus != busn) { + resource_size_t offset; + + devm_iounmap(pci->dev, pci->cfg.base); + offset = pci->cfg.cpu_phys + (busn << 20); + pci->cfg.base = devm_ioremap(pci->dev, offset, SZ_1M);
Quoted 8 lines
drivers/pci/host/Kconfig
Quoted 5 lines
drivers/pci/host/Kconfig +5
-0+ pci->cfg.bus = busn; + } + + return pci->cfg.base + ((devfn << 12) | where); +}Nice idea, but unfortunately broken: we need config space access from atomic context, since there are drivers doing that.
That aside, I just took a spin_lock so this needs fixing regardless. The only solution I can think of then is to map all of the buses at setup time (making bus_ranges pretty important) and hope that I don't chew through all of vmalloc.
Quoted 6 lines
Quoted 6 lines
drivers/pci/host/Kconfig +6
-0+static int gen_pci_probe(struct platform_device *pdev) +{ + struct hw_pci hw; + struct of_pci_range range; + struct of_pci_range_parser parser; + struct gen_pci *pci;
Quoted 8 lines
drivers/pci/host/Kconfig
Quoted 4 lines
drivers/pci/host/Kconfig +4
-0+ const __be32 *reg; + const struct of_device_id *of_id; + struct device *dev = &pdev->dev; + struct device_node *np = dev->of_node;Could you try to move almost all of this function into
gen_pci_setup()? I suspect this will make it easier later to share this driver with other architectures.
I'll take a look. If we get rid of nr_controllers, as suggested later on, the line between probe and setup is somewhat blurred.
Quoted 16 lines
drivers/pci/host/Kconfig
Quoted 12 lines
drivers/pci/host/Kconfig +12
-0+ + hw = (struct hw_pci) { + .nr_controllers = 1, + .private_data = (void **)&pci, + .setup = gen_pci_setup, + .map_irq = of_irq_parse_and_map_pci, + .ops = &gen_pci_ops, + }; + + pci_common_init_dev(dev, &hw); + return 0; +}A missing part here seems to be a way to propagate errors from the
pci_common_init_devorgen_pci_setupback to the caller.
Quoted 11 lines
This seems to be a result of the arm pcibios implementation not being meant for loadable modules, but I suspect it can be fixed. The nr_controllers>1 logic gets a bit in the way there, but it's also a model that seems to be getting out of fashion:
kirkwood/dove/orion5x/mv78xx0 use it at the moment, but are migrating over to the new pci-mvebu code that doesn't as they get rid of the non-DT probing.pci-rcar-gen2.cuses it, but it seems they already ran into problems with that and are changing it. That pretty much leaves iop13xx as the only user, but at that point we can probably move the loop into iop13xx specific code.
Makes sense once there are no users of the field.
Will
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 10 lines · On Thursday 13 February 2014 11:04:02 Will Deacon wrote:
Hi Arnd,
Thanks again for the comments.
Quoted 5 lines · Arnd Bergmann
Quoted 3 lines · On Wednesday 12 February 2014 20:16:11 Will Deacon wrote:
+- ranges : As described in IEEE Std 1275-1994, but must provide
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +2
-0+ at least a definition of one or both of IO and Memory + Space.
Quoted 5 lines
Quoted 2 lines
I'd say *must* provide at least non-prefetchable memory. *may* provide prefetchable memory and/or I/O space.
Can do. Should I enforce this in the driver too? (i.e. complain if non-prefetchable memory is absent).
Yes, good idea.
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/pci/host/Kconfig +0
-0@@ -33,4 +33,11 @@ config PCI_RCAR_GEN2 There are 3 internal PCI controllers available with a single built-in EHCI/OHCI host controller present on each one.Quoted 12 lines
drivers/pci/host/Kconfig config PCI_RCAR_GEN2 · line 43
Quoted 8 lines
drivers/pci/host/Kconfig +7
-0+config PCI_ARM_GENERIC + bool "ARM generic PCI host controller" + depends on ARM && OF + help + Say Y here if you want to support a simple generic PCI host + controller, such as the one emulated by kvmtool. + endmenuLooks good for now. In the long run, I'd hope to get rid of the 'ARM' part here and make it work on any architecture, but that requires significant work that we should not depend on here.
Quoted 2 lines
Agreed. arm64 is the obvious next target (once Liviu's series is sorted out -- I'm currently using a simplified port of
bios32.cfor testing).
See also the reply I just sent on the previous thread for a migration plan regarding the existing drivers.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/host/Kconfig +10
-0+struct gen_pci_cfg_window { + u64 cpu_phys; + void __iomem *base; + u8 bus; + spinlock_t lock; + const struct gen_pci_cfg_accessors *accessors; +}; + +struct gen_pci_resource { + struct list_head list;Quoted 8 lines
drivers/pci/host/Kconfig
Quoted 3 lines
drivers/pci/host/Kconfig +3
-0+ struct resource cpu_res; + resource_size_t offset; +};Your
gen_pci_resourceis actually identical to structpci_host_bridge_window, which I guess is coincidence, but it would be nice to actually use the standard structure to make it easier to integrate with common infrastructure later.
Quoted 2 lines
Ha, at least I managed to come up with the same struct! I'll move to the generic version.
Hmm, I fear I was speaking too quickly, the pci_host_bridge_window actually contains a pointer to the resource rather than the resource itself :(
There is probably a way to do this better, at least once we unify the probe() and setup() functions.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
drivers/pci/host/Kconfig +15
-0+static void __iomem *gen_pci_map_cfg_bus_ecam(struct pci_bus *bus, + unsigned int devfn, + int where) +{ + struct pci_sys_data *sys = bus->sysdata; + struct gen_pci *pci = sys->private_data; + u32 busn = bus->number; + + spin_lock(&pci->cfg.lock); + if (pci->cfg.bus != busn) { + resource_size_t offset; + + devm_iounmap(pci->dev, pci->cfg.base); + offset = pci->cfg.cpu_phys + (busn << 20); + pci->cfg.base = devm_ioremap(pci->dev, offset, SZ_1M);Quoted 8 lines
drivers/pci/host/Kconfig
Quoted 5 lines
drivers/pci/host/Kconfig +5
-0+ pci->cfg.bus = busn; + } + + return pci->cfg.base + ((devfn << 12) | where); +}Nice idea, but unfortunately broken: we need config space access from atomic context, since there are drivers doing that.
Quoted 4 lines
That aside, I just took a
spin_lockso this needs fixing regardless. The only solution I can think of then is to map all of the buses at setup time (makingbus_rangespretty important) and hope that I don't chew through all of vmalloc.
It's possible we have to go further and only map the buses that are actually used, rather than the ones that are defined for the bus, but just mapping the entire range is a reasonable start I think.
Quoted 28 lines
Quoted 16 lines
drivers/pci/host/Kconfig
Quoted 12 lines
drivers/pci/host/Kconfig +12
-0+ + hw = (struct hw_pci) { + .nr_controllers = 1, + .private_data = (void **)&pci, + .setup = gen_pci_setup, + .map_irq = of_irq_parse_and_map_pci, + .ops = &gen_pci_ops, + }; + + pci_common_init_dev(dev, &hw); + return 0; +}A missing part here seems to be a way to propagate errors from the
pci_common_init_devorgen_pci_setupback to the caller.Quoted 11 lines
This seems to be a result of the arm pcibios implementation not being meant for loadable modules, but I suspect it can be fixed. The nr_controllers>1 logic gets a bit in the way there, but it's also a model that seems to be getting out of fashion:
kirkwood/dove/orion5x/mv78xx0 use it at the moment, but are migrating over to the new pci-mvebu code that doesn't as they get rid of the non-DT probing.pci-rcar-gen2.cuses it, but it seems they already ran into problems with that and are changing it. That pretty much leaves iop13xx as the only user, but at that point we can probably move the loop into iop13xx specific code.
Quoted 1 line
Makes sense once there are no users of the field.
With the patch I just suggested, we can simply keep pci_common_init_dev() for older (non-multiplatform) controllers and not change them at all but move on to something else for the interesting ones, i.e. those we want to share with arm64.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 14 lines · Arnd Bergmann
Quoted 13 lines · On Thursday 13 February 2014 11:04:02 Will Deacon wrote:
Quoted 11 lines · Arnd Bergmann
drivers/pci/host/pci-arm-generic.c line 48
Quoted 5 lines · On Wednesday 12 February 2014 20:16:11 Will Deacon wrote:
drivers/pci/host/pci-arm-generic.c new file +5
-0+struct gen_pci_resource { +struct list_head list; +struct resource cpu_res; +resource_size_t offset; +};Your
gen_pci_resourceis actually identical to structpci_host_bridge_window, which I guess is coincidence, but it would be nice to actually use the standard structure to make it easier to integrate with common infrastructure later.
Quoted 5 lines
Quoted 2 lines
Ha, at least I managed to come up with the same struct! I'll move to the generic version.
Hmm, I fear I was speaking too quickly, the
pci_host_bridge_windowactually contains a pointer to the resource rather than the resource itself :(
I can allocate the resources dynamically as I parse them, not a problem at all.
Quoted 2 lines
There is probably a way to do this better, at least once we unify the
probe()andsetup()functions.
Yes, I fully expect this to be iterative.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
drivers/pci/host/pci-arm-generic.c line 97
Quoted 9 lines
drivers/pci/host/pci-arm-generic.c new file +9
-0+spin_lock (&pci->cfg.lock); +if (pci->cfg.bus != busn) { +resource_size_t offset; + +devm_iounmap (pci->dev, pci->cfg.base); + offset = pci->cfg.cpu_phys + (busn <<20 ); + pci->cfg.base =devm_ioremap (pci->dev, offset,SZ_1M ); + pci->cfg.bus = busn; + }Nice idea, but unfortunately broken: we need config space access from atomic context, since there are drivers doing that.
Quoted 8 lines
Quoted 4 lines
That aside, I just took a
spin_lockso this needs fixing regardless. The only solution I can think of then is to map all of the buses at setup time (makingbus_rangespretty important) and hope that I don't chew through all of vmalloc.It's possible we have to go further and only map the buses that are actually used, rather than the ones that are defined for the bus, but just mapping the entire range is a reasonable start I think.
Okey doke.
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
drivers/pci/host/pci-arm-generic.c line 304
Quoted 11 lines
drivers/pci/host/pci-arm-generic.c new file +11
-0+ hw = (struct hw_pci ) { + .nr_controllers =1 , + .private_data = (void **)&pci, + .setup = gen_pci_setup, + .map_irq = of_irq_parse_and_map_pci, + .ops = &gen_pci_ops, + }; + +pci_common_init_dev (dev, &hw); +return 0 ; +}A missing part here seems to be a way to propagate errors from the
pci_common_init_devorgen_pci_setupback to the caller.This seems to be a result of the arm pcibios implementation not being meant for loadable modules, but I suspect it can be fixed. The nr_controllers>1 logic gets a bit in the way there, but it's also a model that seems to be getting out of fashion:
kirkwood/dove/orion5x/mv78xx0 use it at the moment, but are migrating over to the new pci-mvebu code that doesn't as they get rid of the non-DT probing.pci-rcar-gen2.cuses it, but it seems they already ran into problems with that and are changing it. That pretty much leaves iop13xx as the only user, but at that point we can probably move the loop into iop13xx specific code.
Quoted 6 lines
Quoted 1 line
Makes sense once there are no users of the field.
With the patch I just suggested, we can simply keep
pci_common_init_dev()for older (non-multiplatform) controllers and not change them at all but move on to something else for the interesting ones, i.e. those we want to share with arm64.
Yes, that looks sensible. An alternative is to create an hw_pci in pci_host_bridge_register with nr_controllers = 1 then call pci_common_init_dev. It would remove slightly more code, but obviously ties the thing to arm.
Will
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 9 lines · On Thursday 13 February 2014 12:00:42 Will Deacon wrote:
Quoted 4 lines
With the patch I just suggested, we can simply keep
pci_common_init_dev()for older (non-multiplatform) controllers and not change them at all but move on to something else for the interesting ones, i.e. those we want to share with arm64.Yes, that looks sensible. An alternative is to create an
hw_pciinpci_host_bridge_registerwithnr_controllers= 1 then callpci_common_init_dev. It would remove slightly more code, but obviously ties the thing to arm.
You'd still need to pass all the contents of the hw_pci struct that get copied into pci_sys_data, so that's not different from passing hw_pci as we do today.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Will DeaconQuoted 32 lines · Will Deacon
This patch adds support for a generic PCI host controller, such as a firmware-initialised device with static windows or an emulation by something such as kvmtool.
The controller itself has no configuration registers and has its address spaces described entirely by the device-tree (using the bindings from ePAPR). Both CAM and ECAM are supported for Config Space accesses.
Corresponding documentation is added for the DT binding.
Signed-off-by Will Deacon <[email protected]>---
.../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 4 files changed, 377 insertions(+)create mode 100644
Documentation/devicetree/bindings/pci/arm-generic-pci.txtcreate mode 100644drivers/pci/host/pci-arm-generic.cDocumentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0@@ -0,0 +1,51 @@ +* ARM generic PCI host controller + +Firmware-initialised PCI host controllers and PCI emulations, such as the +virtio-pci implementations found in kvmtool and other para-virtualised +systems, do not require driver support for complexities such as regulator and +clock management. In fact, the controller may not even require the
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 16
Quoted 10 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +10
-0+configuration of a control interface by the operating system, instead +presenting a set of fixed windows describing a subset of IO, Memory and +Configuration Spaces. + +Such a controller can be described purely in terms of the standardized device +tree bindings communicated in pci.txt: + +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively)
What?s arm specific here? I don?t have a great suggestion, but seems odd for this to be vendor prefixed with "arm".
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 27
Quoted 11 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +11
-0+ +- ranges : As described in IEEE Std 1275-1994, but must provide + at least a definition of one or both of IO and Memory + Space. + +- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus.
Isn?t the size fixed here for cam or ecam?
Quoted 14 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +14
-0+ +Configuration Space is assumed to be memory-mapped (as opposed to being +accessed via an ioport) and laid out with a direct correspondence to the +geography of a PCI bus address by concatenating the various components to form +an offset. + +For CAM, this 24-bit offset is: + + cfg_offset(bus, device, function, register) = + bus << 16 | device << 11 | function << 8 | register + +Whilst ECAM extends this by 4 bits to accomodate 4k of function space: + + cfg_offset(bus, device, function, register) =
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 51
Quoted 10 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +10
-0+ bus << 20 | device << 15 | function << 12 | register + +Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties: + +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>
Examples are always nice :)
- k
Signature · 2 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 40 lines · Kumar Gala
Quoted 37 lines · Will Deacon
This patch adds support for a generic PCI host controller, such as a firmware-initialised device with static windows or an emulation by something such as kvmtool.
The controller itself has no configuration registers and has its address spaces described entirely by the device-tree (using the bindings from ePAPR). Both CAM and ECAM are supported for Config Space accesses.
Corresponding documentation is added for the DT binding.
Signed-off-by Will Deacon <[email protected]>---
.../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 4 files changed, 377 insertions(+)create mode 100644
Documentation/devicetree/bindings/pci/arm-generic-pci.txtcreate mode 100644drivers/pci/host/pci-arm-generic.cDocumentation/devicetree/bindings/pci/arm-generic-pci.txt new file +11
-0@@ -0,0 +1,51 @@ +* ARM generic PCI host controller + +Firmware-initialised PCI host controllers and PCI emulations, such as the +virtio-pci implementations found in kvmtool and other para-virtualised +systems, do not require driver support for complexities such as regulator and +clock management. In fact, the controller may not even require the +configuration of a control interface by the operating system, instead +presenting a set of fixed windows describing a subset of IO, Memory and +Configuration Spaces. + +Such a controller can be described purely in terms of the standardized device
Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 16
Quoted 5 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +5
-0+tree bindings communicated in pci.txt: + +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively)What?s arm specific here? I don?t have a great suggestion, but seems odd for this to be vendor prefixed with "arm".
Happy to change it, but I'm also struggling for names. Maybe "linux,..."?
Quoted 4 lines
Quoted 4 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +4
-0+- ranges : As described in IEEE Std 1275-1994, but must provide + at least a definition of one or both of IO and Memory + Space. +
Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 27
Quoted 6 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0+- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus.Isn?t the size fixed here for cam or ecam?
Yes, which is why reg just specifies the base address.
Quoted 17 lines
Quoted 17 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +17
-0+Configuration Space is assumed to be memory-mapped (as opposed to being +accessed via an ioport) and laid out with a direct correspondence to the +geography of a PCI bus address by concatenating the various components to form +an offset. + +For CAM, this 24-bit offset is: + + cfg_offset(bus, device, function, register) = + bus << 16 | device << 11 | function << 8 | register + +Whilst ECAM extends this by 4 bits to accomodate 4k of function space: + + cfg_offset(bus, device, function, register) = + bus << 20 | device << 15 | function << 12 | register + +Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties:
Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 51
Quoted 6 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0+ +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>Examples are always nice :)
Not in this case! kvmtool generates the following:
pci {
#address-cells = <0x3>;
#size-cells = <0x2>;
#interrupt-cells = <0x1>;
compatible = "arm,pci-cam-generic";
reg = <0x0 0x40000000>;
ranges = <0x1000000 0x0 0x0 0x0 0x0 0x0 0x10000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>;
interrupt-map = <0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1 0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1 0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1 0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>;
interrupt-map-mask = <0xf800 0x0 0x0 0x7>;
};I can add it if you like, but it looks like a random bunch of numbers to me.
Will
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 50 lines · Will Deacon
Quoted 40 lines · Kumar Gala
Quoted 37 lines · Will Deacon
This patch adds support for a generic PCI host controller, such as a firmware-initialised device with static windows or an emulation by something such as kvmtool.
The controller itself has no configuration registers and has its address spaces described entirely by the device-tree (using the bindings from ePAPR). Both CAM and ECAM are supported for Config Space accesses.
Corresponding documentation is added for the DT binding.
Signed-off-by Will Deacon <[email protected]>---
.../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 4 files changed, 377 insertions(+)create mode 100644
Documentation/devicetree/bindings/pci/arm-generic-pci.txtcreate mode 100644drivers/pci/host/pci-arm-generic.cDocumentation/devicetree/bindings/pci/arm-generic-pci.txt new file +11
-0@@ -0,0 +1,51 @@ +* ARM generic PCI host controller + +Firmware-initialised PCI host controllers and PCI emulations, such as the +virtio-pci implementations found in kvmtool and other para-virtualised +systems, do not require driver support for complexities such as regulator and +clock management. In fact, the controller may not even require the +configuration of a control interface by the operating system, instead +presenting a set of fixed windows describing a subset of IO, Memory and +Configuration Spaces. + +Such a controller can be described purely in terms of the standardized deviceQuoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 16
Quoted 5 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +5
-0+tree bindings communicated in pci.txt: + +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively)What?s arm specific here? I don?t have a great suggestion, but seems odd for this to be vendor prefixed with "arm".
Quoted 1 line
Happy to change it, but I'm also struggling for names. Maybe "linux,?"?
I was thinking that as well, I?d say go with ?linux,?.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +4
-0+- ranges : As described in IEEE Std 1275-1994, but must provide + at least a definition of one or both of IO and Memory + Space. +Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 27
Quoted 6 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0+- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus.Isn?t the size fixed here for cam or ecam?
Quoted 1 line
Yes, which is why reg just specifies the base address.
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
Quoted 40 lines
Quoted 18 lines
Quoted 18 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +17
-0+Configuration Space is assumed to be memory-mapped (as opposed to being +accessed via an ioport) and laid out with a direct correspondence to the +geography of a PCI bus address by concatenating the various components to form +an offset. + +For CAM, this 24-bit offset is: + + cfg_offset(bus, device, function, register) = + bus << 16 | device << 11 | function << 8 | register + +Whilst ECAM extends this by 4 bits to accomodate 4k of function space: + + cfg_offset(bus, device, function, register) = + bus << 20 | device << 15 | function << 12 | register + +Interrupt mapping is exactly as described in `Open Firmware Recommended +Practice: Interrupt Mapping' and requires the following properties:Quoted 8 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 51
Quoted 6 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0+ +- #interrupt-cells : Must be 1 + +- interrupt-map : <see aforementioned specification> + +- interrupt-map-mask : <see aforementioned specification>Examples are always nice :)
Not in this case! kvmtool generates the following:
pci { #address-cells = <0x3>; #size-cells = <0x2>; #interrupt-cells = <0x1>; compatible = "arm,pci-cam-generic"; reg = <0x0 0x40000000>; ranges = <0x1000000 0x0 0x0 0x0 0x0 0x0 0x10000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>; interrupt-map = <0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1 0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1 0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1 0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>; interrupt-map-mask = <0xf800 0x0 0x0 0x7>; };
Quoted 1 line
I can add it if you like, but it looks like a random bunch of numbers to me.
You could clean it up a bit to be human readable even if its kvmtool that?s creating it.
pci {
compatible = "arm,pci-cam-generic?;
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>
reg = <0x0 0x40000000>;
ranges = <
0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000
0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000
>;
interrupt-map = <
...
>;
interrupt-map-mask = <0xf800 0x0 0x0 0x7>;Quoted 2 lines
Will
- k
Signature · 2 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 6 lines · Kumar Gala
Quoted 5 lines · Will Deacon
Quoted 5 lines
Quoted 4 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +4
-0+ +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively)
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
What?s arm specific here? I don?t have a great suggestion, but seems odd for this to be vendor prefixed with "arm".
Happy to change it, but I'm also struggling for names. Maybe "linux,?"?
I was thinking that as well, I?d say go with ?linux,?.
Great, I'll make that change.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 27
Quoted 10 lines
+- ranges : As described in IEEE Std 1275-1994, but must provide
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +9
-0+ at least a definition of one or both of IO and Memory + Space. + +- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus.Isn?t the size fixed here for cam or ecam?
Quoted 5 lines
Quoted 1 line
Yes, which is why reg just specifies the base address.
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
Actually, the example just has a 64-bit CPU physical address with no size. Do I have to add a size to the binding? It's not at all useful for the driver.
Quoted 19 lines
Quoted 15 lines
Quoted 1 line
Examples are always nice :)
Not in this case! kvmtool generates the following:
pci { #address-cells = <0x3>; #size-cells = <0x2>; #interrupt-cells = <0x1>; compatible = "arm,pci-cam-generic"; reg = <0x0 0x40000000>; ranges = <0x1000000 0x0 0x0 0x0 0x0 0x0 0x10000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>; interrupt-map = <0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1 0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1 0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1 0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>; interrupt-map-mask = <0xf800 0x0 0x0 0x7>; };Quoted 1 line
I can add it if you like, but it looks like a random bunch of numbers to me.
You could clean it up a bit to be human readable even if its kvmtool that?s creating it.
Quoted 15 lines
pci {
compatible = "arm,pci-cam-generic?;
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>
reg = <0x0 0x40000000>;
ranges = <
0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000
0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000
>;
interrupt-map = <
...
>;
interrupt-map-mask = <0xf800 0x0 0x0 0x7>;Sure, if you think it helps.
Will
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Kumar GalaQuoted 4 lines · On Thursday 13 February 2014 10:22:25 Kumar Gala wrote:
Quoted 1 line · Will Deacon
Happy to change it, but I'm also struggling for names. Maybe "linux,?"?
I was thinking that as well, I?d say go with ?linux,?.
I see nothing linux specific in there. I would only use that namespace for things we don't expect to work with another OS.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt line 27
Quoted 10 lines
+- ranges : As described in IEEE Std 1275-1994, but must provide
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +9
-0+ at least a definition of one or both of IO and Memory + Space. + +- #address-cells : Must be 3 + +- #size-cells : Must be 2 + +- reg : The Configuration Space base address, as accessed by the + parent bus.Isn?t the size fixed here for cam or ecam?
Quoted 5 lines
Quoted 1 line
Yes, which is why reg just specifies the base address.
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
It's still an open question whether the config space in the reg property should cover all 256 buses or just the ones in the bus-range. In the latter case, it would be variable (but predictable) size.
Quoted 12 lines
You could clean it up a bit to be human readable even if its kvmtool that?s creating it.
pci { compatible = "arm,pci-cam-generic?; #address-cells = <3>; #size-cells = <2>; #interrupt-cells = <1> reg = <0x0 0x40000000>; ranges = < 0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000 >;
Make it
ranges = <0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000>,
<0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>;
and it will be perfect ;-)
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
[adding devicetree, Grant, Rob]
Apologies to all those who aren't too bothered about the naming, I'm going to derail this subthread for a general device tree policy discussion.
Quoted 8 lines · Arnd Bergmann
Quoted 4 lines · On Thursday 13 February 2014 10:22:25 Kumar Gala wrote:
Quoted 1 line · Will Deacon
Happy to change it, but I'm also struggling for names. Maybe "linux,?"?
I was thinking that as well, I?d say go with ?linux,?.
I see nothing linux specific in there. I would only use that namespace for things we don't expect to work with another OS.
Let's take a stop back. This is an instance of a wider problem, and to be honest it's a rather trivial one. As usual, because there's been no policy there's no consistency, and everyone's got their own favourite.
All we need is a policy for how generic bindings are named/prefixed. As long as this is consistent and unlikely to cause problems it doesn't really matter what that specific policy is.
Regardless, we need to support our existing bindings.
So far we have instances of (in v3.14-rc2):
A) No prefix
$ git grep 'compatible\s*=\s*"[^,]*"' -- Documentation/devicetree | \
awk '{ print $NF }' | sort -u | wc -l
96 $ git grep 'compatible\s*=\s*"[^,]*"' -- arch/*/boot/dts | \
awk ' { print $NF }' | sort -u | wc -l
233These include some commonly use bindings like "fixed-clock", "regulator-fixed", and (everybody's favourite) "simple-bus". The "ns16550" binding falls here too.
There are subdivisions within this, like "simple-*" and "*-generic". There are three "simple-" prefixed variants, perhaps that's worth it's own class?
There are also some dubious ones like "ldo4" that are probably only appearing in sub-nodes and probably don't matter that much for this discussion.
B) "linux," prefixed
$ git grep 'compatible\s*=\s*"linux,.*"' -- Documentation/devicetree | \
awk ' { print $NF }' | sort -u | wc -l
4 $ git grep 'compatible\s*=\s*"linux,.*"' -- arch/*/boot/dts | \
awk ' {print $NF }' | sort -u | wc -l
1 These include:
* "linux,hdmi-audio"
* "linux,spdif-dir"
* "linux,spdif-dit"
* "linux,wdt-gpio"
Is there another class I've missed?
If a binding originates for Linux, a "linux," prefix doesn't seem that bad to me. It's just a namespace, it doesn't have to mean that it's only ever going to work for Linux, and two such bindings within Linux shouldn't be able to collide.
As long as we choose something that reduces the possibility of name collisions (several people might come up with incompatible "pci" bindings) then we're fine regardless of the particular prefix or lack thereof. A "generic," prefix feels like it's too easy to collide with if another community also creates bindings.
Thoughts?
One final option for generic bindings would be a prefix per-author (e.g. in this case the compatible string could be "will-deacon,pci"). That might lead to the fewest namespace issues in future, and helps to immortalise/demonise the original author of any binding claiming to be generic. Patch below.
Thanks,
Mark.
---->8----
Quoted 1 line
From
6ed3e3564199ed6dbd8782a740d4067fb1e6d3b6Mon Sep 17 00:00:00 2001
From: Mark Rutland <[email protected]>
Date: Thu, 13 Feb 2014 17:50:33 +0000
Subject: [PATCH] Documentation: devicetree: add vendor prefix for Will Deacon
This patch adds a vendor prefix for Will Deacon, for use in generic hardware bindings originating from Will Deacon.
--- Documentation/devicetree/bindings/vendor-prefixes.txt | 1 + 1 file changed, 1 insertion(+)
Documentation/devicetree/bindings/vendor-prefixes.txt +1 -0
@@ -88,6 +88,7 @@ toumaz Toumaz v3 V3 Semiconductor via VIA Technologies, Inc. winbond Winbond Electronics corp. +will-deacon Will Deacon MEng wlf Wolfson Microelectronics wm Wondermedia Technologies, Inc. xlnx Xilinx
Signature · 1 line
1.8.1.1
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Arnd BergmannQuoted 9 lines · Arnd Bergmann
Quoted 4 lines
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
It's still an open question whether the config space in the reg property should cover all 256 buses or just the ones in the bus-range. In the latter case, it would be variable (but predictable) size.
The 'describe the hardware principle' says the reg should be the entire available ECAM/CAM region the hardware is able to support.
This may be less than 256 busses, as ECAM allows the implementor to select how many upper address bits are actually supported.
IMHO, the bus-range should be used to indicate the range of busses discovered by the firmware, but we have historically tweaked it to indicate the max range of bus numbers available on this bus (I think to support the hack where two physical PCI domains were roughly glued into a single Linux domain).
Which is not necessary when the DT stanza maps 1:1 into a PCI domain. The driver default for bus-range should just be 0 to 255, and it shouldn't be included in most cases.
The max busnr resource passed to the PCI core should be the lower of the busnr property and the reg limit, IMHO.
The issue with burning VM in Linux is a Linux issue and shouldn't leak into the DT... Ideally the solution here would be to have the PCI core call back to the host driver when it allocates/deallocates a bus number and then the driver can manage the config space VM mapping on a bus by bus basis.
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
It's still an open question whether the config space in the reg property should cover all 256 buses or just the ones in the bus-range. In the latter case, it would be variable (but predictable) size.
Quoted 5 lines
The 'describe the hardware principle' says the reg should be the entire available ECAM/CAM region the hardware is able to support.
This may be less than 256 busses, as ECAM allows the implementor to select how many upper address bits are actually supported.
Ok, but the ECAM/CAM base always corresponds to bus 0, right?
Quoted 5 lines
IMHO, the bus-range should be used to indicate the range of busses discovered by the firmware, but we have historically tweaked it to indicate the max range of bus numbers available on this bus (I think to support the hack where two physical PCI domains were roughly glued into a single Linux domain).
Ok, so this answers Kumar's point about the reg property. I'll augment it with a size.
Will
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 4 lines · Will Deacon
Quoted 2 lines
This may be less than 256 busses, as ECAM allows the implementor to select how many upper address bits are actually supported.
Ok, but the ECAM/CAM base always corresponds to bus 0, right?
Yes, or it isn't ECAM.
Quoted 2 lines
Ok, so this answers Kumar's point about the reg property. I'll augment it with a size.
Don't forget to request_region it as well...
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Will DeaconQuoted 13 lines · On Thursday 13 February 2014 19:53:17 Will Deacon wrote:
Quoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
It's still an open question whether the config space in the reg property should cover all 256 buses or just the ones in the bus-range. In the latter case, it would be variable (but predictable) size.
Quoted 7 lines
Quoted 5 lines
The 'describe the hardware principle' says the reg should be the entire available ECAM/CAM region the hardware is able to support.
This may be less than 256 busses, as ECAM allows the implementor to select how many upper address bits are actually supported.
Ok, but the ECAM/CAM base always corresponds to bus 0, right?
Ah, plus I suppose it ought to be a power-of-two size?
Quoted 5 lines
Quoted 5 lines
IMHO, the bus-range should be used to indicate the range of busses discovered by the firmware, but we have historically tweaked it to indicate the max range of bus numbers available on this bus (I think to support the hack where two physical PCI domains were roughly glued into a single Linux domain).
There is an interesting point about the domain assignment, brought to my attention by Russell's comment about the hw_pci struct: If we want to support arbitrary combinations of pci host bridges described in DT, we need a better policy to decide what domain to use. The approaches I've seen so far are:
1. We assume each host bridge in DT is a domain by itself. I think we do that for all DT probed bridges on ARM (aside from shmobile) at the moment. In some cases, the the host bridge is a really a fiction made up by the host driver to couple various identical but independent PCIe root ports, but the same fiction is shared between DT and the PCI core view of it. This requires that we enable the PCI domain code unconditionally, and breaks all user space that doesn't understand domains (this should be rare but can still exist for x86 based software).
2. The architecture or platform code decides and uses a method equivalent to ARM's pci_common_init_dev() after it has found all host bridges. The architecture "knows" how many domains it wants and calls pci_common_init_dev() for each domain, and then the setup() callbacks grab as many buses as they need within the domain. For a generic multiplatform kernel, this means we need to add a top-level driver that looks at all pci hosts in DT before any of them are probed. It also means the pci host drivers can't be loadable modules.
3. We assume there is only one domain, and require each host bridge in DT to specify a bus-range that is a subset of the available 256 bus numbers. This should work for anything but really big systems with many hot-pluggable ports, since we need to reserve a few bus numbers on each port for hotplugging.
4. Like 3, but start a new domain if the bus-range properties don't fit in the existing domains.
5. Like 3, but specify a generic "pci-domain" property for DT that allows putting host bridges into explicit domains in a predictable way.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 61 lines · Arnd Bergmann
Quoted 13 lines · On Thursday 13 February 2014 19:53:17 Will Deacon wrote:
Quoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines
Huh? The reg property clearly has the size in it (as shown in the example below). I guess I was just asking for the description here to say what the size was for the 2 compatibles since its fixed and known.
It's still an open question whether the config space in the reg property should cover all 256 buses or just the ones in the bus-range. In the latter case, it would be variable (but predictable) size.
Quoted 7 lines
Quoted 5 lines
The 'describe the hardware principle' says the reg should be the entire available ECAM/CAM region the hardware is able to support.
This may be less than 256 busses, as ECAM allows the implementor to select how many upper address bits are actually supported.
Ok, but the ECAM/CAM base always corresponds to bus 0, right?
Ah, plus I suppose it ought to be a power-of-two size?
Quoted 5 lines
Quoted 5 lines
IMHO, the bus-range should be used to indicate the range of busses discovered by the firmware, but we have historically tweaked it to indicate the max range of bus numbers available on this bus (I think to support the hack where two physical PCI domains were roughly glued into a single Linux domain).
There is an interesting point about the domain assignment, brought to my attention by Russell's comment about the
hw_pcistruct: If we want to support arbitrary combinations of pci host bridges described in DT, we need a better policy to decide what domain to use. The approaches I've seen so far are:1. We assume each host bridge in DT is a domain by itself. I think we do that for all DT probed bridges on ARM (aside from shmobile) at the moment. In some cases, the the host bridge is a really a fiction made up by the host driver to couple various identical but independent PCIe root ports, but the same fiction is shared between DT and the PCI core view of it. This requires that we enable the PCI domain code unconditionally, and breaks all user space that doesn't understand domains (this should be rare but can still exist for x86 based software).
2. The architecture or platform code decides and uses a method equivalent to ARM's
pci_common_init_dev()after it has found all host bridges. The architecture "knows" how many domains it wants and callspci_common_init_dev()for each domain, and then thesetup()callbacks grab as many buses as they need within the domain. For a generic multiplatform kernel, this means we need to add a top-level driver that looks at all pci hosts in DT before any of them are probed. It also means the pci host drivers can't be loadable modules.3. We assume there is only one domain, and require each host bridge in DT to specify a bus-range that is a subset of the available 256 bus numbers. This should work for anything but really big systems with many hot-pluggable ports, since we need to reserve a few bus numbers on each port for hotplugging.
Quoted 6 lines
4. Like 3, but start a new domain if the bus-range properties don't fit in the existing domains.
5. Like 3, but specify a generic "pci-domain" property for DT that allows putting host bridges into explicit domains in a predictable way.
What I'm going to suggest in my v2 patch (hope to send it before Monday) is a new API in the generic PCI code that will allow you to create a host bridge in a new domain or in the existing domain, with the management of the domain number being done in the generic code.
Something like:
int create_hostbridge_in_new_domain(....); int create_hostbridge(....);
with the functions being wrappers around the pci_hostbridge_of_init function that I'm introducing.
What do you think?
Best regards,
Liviu
Quoted 7 lines
Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 8 lines · On Friday 14 February 2014 22:00:47 Liviu Dudau wrote:
What I'm going to suggest in my v2 patch (hope to send it before Monday) is a new API in the generic PCI code that will allow you to create a host bridge in a new domain or in the existing domain, with the management of the domain number being done in the generic code.
Something like:
Quoted 7 lines
int create_hostbridge_in_new_domain(....); int create_hostbridge(....);with the functions being wrappers around the
pci_hostbridge_of_initfunction that I'm introducing.What do you think?
Not sure. It would still keep the decision about whether to use a new domain or not in the host bridge driver, but that doesn't seem like the right place. The same driver might be used in different ways based on what is around it.
I've also had a look at the MIPS implementation now, which has its own way of dealing with this, see arch/mips/pci/pci.c.
There was also an interesting comment in the MIPS header file:
/* For compatibility with current (as of July 2003) pciutils
and XFree86. Eventually will be removed. */
unsigned int need_domain_info;This is over ten years old, so I wonder if we can start assuming that domains work out of the box now. All references to problems from PCI domains are about old code (ca. pre-2007) that doesn't understand nonzero domains and that has since been fixed. I am pretty sure we don't need to ever worry about stuffing multiple host bridges into a domain other than the first one, and I also doubt we have to worry about the problem at all on arm64 as we won't run old binaries on it (or arm32 compat mode binaries that need to manage PCI devices).
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
If we do need to stuff multiple buses into one domain, how about using this approach in pci_hostbridge_of_init():
The first time we add a host bridge, scan the entire DT for devices setting device_type="pci". If there is actually more than one host bridge, check all "bus-range" properties to see if they overlap. If there is no overlap or only one bridge, don't use domains.
If there is any overlap at all, or there are multiple host bridge and one of them does not have a bus-range property, use a separate domain per host bridge.
Remember the function was called before so it doesn't have to scan the DT again, and count the domain locally.
Does this make sense?
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 3 lines · Arnd Bergmann
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
I personaly think we can safely move away from stuffing multiple host bridges into a single domain for DT cases. The reasons for doing this have long since been superseded.
Most importantly, I have a feeling keeping a 1:1 relationship between domain and driver will making building a proper modular and hot pluggable host driver infrastructure in the PCI core significantly simpler. The domain object gives a nice natural place to put things in sysfs, a natural place to keep function pointers and avoids all the messy questions of what happens if probing overlaps bus numbers, how do you number things, how do you hot plug downstream busses, etc.
Having a PCI core driver infrastructure that supports both 'as a domain' and 'as a bunch of busses' seems much more complex, and I can't really identify what is being gained by continuing to support this.
As far as I know the host bridge stuffing is something that was created before domains to solve the problem on embedded of multiple PCI host bridges on a SOC/System Controller. I know I have used it that way in the distant past (for MIPS).
However today PCI-SIG has a standard way to describe multi-port root-complexes in config space, so we should not need to use the multi-bus hack. SOCs with non-compliant HW that *really* need single domain can get there: mvebu shows how to write a driver that provides a software version of the missing hardware elements. Pushing mess like this out of the core code seems like a good strategy.
The only reason I can think of to avoid using a domain is if Linux has to interface with external firmware that uses bus:device.function notation for coding information. (eg Intel-MP tables on x86 encode interrupt routing use B:D.F) In this case Linux would need a way to map between Linux B:D.F and the Firwmare B:D.F, or it would need to use the Firmware B:D.F layout. But this argument does not apply to DT systems as DT encodes the domain too. Presumably ACPI will be the same.
Also, bear in mind we now already have multi-domain host drivers for ARM, so the multi-platform kernels need to have this option turned on.
So the Liviu, I would say the API should be similar to what we see in other OF enabled driver bases subsystems, call the core code with a platform_device pointer and function_ops pointer and have the core code create a domain, figure out the domain # from the DT (via aliases?), and so on.
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 9 lines · On Tuesday 18 February 2014 10:41:25 Jason Gunthorpe wrote:
Quoted 3 lines · Arnd Bergmann
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
I personaly think we can safely move away from stuffing multiple host bridges into a single domain for DT cases. The reasons for doing this have long since been superseded.
Ok, that would definitely be the simplest answer. :)
Quoted 4 lines
As far as I know the host bridge stuffing is something that was created before domains to solve the problem on embedded of multiple PCI host bridges on a SOC/System Controller. I know I have used it that way in the distant past (for MIPS).
Apple have also used the same trick on their G5 Macs, presumably to simplify things for OS9 and OS-X, but even at the time making it harder for Linux.
Quoted 6 lines
However today PCI-SIG has a standard way to describe multi-port root-complexes in config space, so we should not need to use the multi-bus hack. SOCs with non-compliant HW that *really* need single domain can get there: mvebu shows how to write a driver that provides a software version of the missing hardware elements. Pushing mess like this out of the core code seems like a good strategy.
Ok.
Quoted 8 lines
The only reason I can think of to avoid using a domain is if Linux has to interface with external firmware that uses bus:device.function notation for coding information. (eg Intel-MP tables on x86 encode interrupt routing use B:D.F) In this case Linux would need a way to map between Linux B:D.F and the Firwmare B:D.F, or it would need to use the Firmware B:D.F layout. But this argument does not apply to DT systems as DT encodes the domain too. Presumably ACPI will be the same.
ACPI definitely has a concept of domains, as I noticed when looking at the x86 ACPI PCI probing code.
Quoted 5 lines
So the Liviu, I would say the API should be similar to what we see in other OF enabled driver bases subsystems, call the core code with a
platform_devicepointer andfunction_opspointer and have the core code create a domain, figure out the domain # from the DT (via aliases?), and so on.
Do we even need stable domain numbers? If we do, aliases sound fine. A more complex method would be to sort them by MMIO window address or perhaps by phandle.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 3 lines · Arnd Bergmann
Apple have also used the same trick on their G5 Macs, presumably to simplify things for OS9 and OS-X, but even at the time making it harder for Linux.
I actually think some x86's have done the same thing, however the firmware hides all that from the OS and the OS just sees a normal PCI environment. So if we can't hide the mess in firmware, the next best place is in drivers?
Quoted 3 lines
Do we even need stable domain numbers? If we do, aliases sound fine. A more complex method would be to sort them by MMIO window address or perhaps by phandle.
PCI ordering has been a bane in the past on x86, so I think stable domain numbers is certainly desirable. Particularly since the domain number may be influenced by module load order :( Alises followed by sorting by phandle sounds great to me.
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 9 lines · On Tuesday 18 February 2014 11:45:42 Jason Gunthorpe wrote:
Quoted 4 lines
Do we even need stable domain numbers? If we do, aliases sound fine. A more complex method would be to sort them by MMIO window address or perhaps by phandle.
PCI ordering has been a bane in the past on x86, so I think stable domain numbers is certainly desirable. Particularly since the domain number may be influenced by module load order Alises followed by sorting by phandle sounds great to me.
Ok, makes sense. Let's make sure we come up with a stable numbering enforced by DT then, and warn when it's not there (or refuse to work even).
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Jason GunthorpeQuoted 44 lines · Jason Gunthorpe
Quoted 3 lines · Arnd Bergmann
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
I personaly think we can safely move away from stuffing multiple host bridges into a single domain for DT cases. The reasons for doing this have long since been superseded.
Most importantly, I have a feeling keeping a 1:1 relationship between domain and driver will making building a proper modular and hot pluggable host driver infrastructure in the PCI core significantly simpler. The domain object gives a nice natural place to put things in sysfs, a natural place to keep function pointers and avoids all the messy questions of what happens if probing overlaps bus numbers, how do you number things, how do you hot plug downstream busses, etc.
Having a PCI core driver infrastructure that supports both 'as a domain' and 'as a bunch of busses' seems much more complex, and I can't really identify what is being gained by continuing to support this.
As far as I know the host bridge stuffing is something that was created before domains to solve the problem on embedded of multiple PCI host bridges on a SOC/System Controller. I know I have used it that way in the distant past (for MIPS).
However today PCI-SIG has a standard way to describe multi-port root-complexes in config space, so we should not need to use the multi-bus hack. SOCs with non-compliant HW that *really* need single domain can get there: mvebu shows how to write a driver that provides a software version of the missing hardware elements. Pushing mess like this out of the core code seems like a good strategy.
The only reason I can think of to avoid using a domain is if Linux has to interface with external firmware that uses bus:device.function notation for coding information. (eg Intel-MP tables on x86 encode interrupt routing use B:D.F) In this case Linux would need a way to map between Linux B:D.F and the Firwmare B:D.F, or it would need to use the Firmware B:D.F layout. But this argument does not apply to DT systems as DT encodes the domain too. Presumably ACPI will be the same.
Quoted 8 lines
Also, bear in mind we now already have multi-domain host drivers for ARM, so the multi-platform kernels need to have this option turned on.
So the Liviu, I would say the API should be similar to what we see in other OF enabled driver bases subsystems, call the core code with a
platform_devicepointer andfunction_opspointer and have the core code create a domain, figure out the domain # from the DT (via aliases?), and so on.
I wish things were easier!
Lets look at the 'int pci_domain_nr(struct pci_bus *bus);' function. It is used to obtain the domain number of the bus passed as an argument.
- include/linux/pci.h defines it as an inline function returning zero if !CONFIG_PCI || !CONFIG_PCI_DOMAINS. Otherwise it is silent on what the function might look like.
- alpha, ia64, microblaze, mips, powerpc and tile all define it as a cast of bus->sysdata to "struct pci_controller *" and then access a data member from there to get the domain number. But ... the pci_controller structure is different for each architecture, with few more members in addition that might be actually shared with generic framework code.
- arm, s390, sparc and x86 have all different names for their sysdata, but they all contain inside a member that is used for giving a domain back. sparc gets an honorary mention here for getting the API wrong and returning -ENXIO in certain conditions (not like the generic code cares).
That takes care of the implementation. But what about usage?
- drivers/pci/probe.c: pci_create_root_bus allocates a new bus structure, sets up the sysdata and ops member pointers and then goes straight into trying to find if the newly created bus doesn't already exist by using the bus number given as parameter and pci_domain_nr() with the new bus structure as argument. I'm trying really hard to figure out what was the intention here, but from the point of view of someone trying to implement the pci_domain_nr() function I have no idea what to return for a virgin bus structure that is not yet attached to any parent.
So I can see the intent of what Jason is proposing and I'm heading that way myself, but I think I need to cleanup pci_create_root_bus first (change the creation order between bridge and bus). And if someone has a good idea on how to determine the domain # from DT we can pluck it into the pcibios_root_bridge_prepare() function (either the generic version or the arch specific one).
Best regards,
Liviu
Quoted 3 lines
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 1 line · Liviu Dudau
I wish things were easier!
Yah, I've seen some of this complexity too..
A thought that crosses my head is to try and leave the entire arch support intact and build a separate 'domain-driver-based' support that completely replaces the arch support, a domain would use one or the other but never both.
So pci_domain_nr could become:
struct pci_domain_driver *drv = pci_get_domain_driver(bus);
if (drv)
return drv->domain_nr;
else
return _arch_pci_domain_nr(bus);And other funcs would change effectively the same way.
Quoted 3 lines
And if someone has a good idea on how to determine the domain # from DT we can pluck it into the
pcibios_root_bridge_prepare()function (either the generic version or the arch specific one).
You can probably start with 'of_alias_get_id'..
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Liviu DudauQuoted 8 lines · On Wednesday 19 February 2014 02:44:27 Liviu Dudau wrote:
Quoted 5 lines · Jason Gunthorpe
So the Liviu, I would say the API should be similar to what we see in other OF enabled driver bases subsystems, call the core code with a
platform_devicepointer andfunction_opspointer and have the core code create a domain, figure out the domain # from the DT (via aliases?), and so on.
Quoted 8 lines
I wish things were easier!
Lets look at the 'int
pci_domain_nr(struct pci_bus *bus);' function. It is used to obtain the domain number of the bus passed as an argument.-
include/linux/pci.hdefines it as an inline function returning zero if !CONFIG_PCI || !CONFIG_PCI_DOMAINS. Otherwise it is silent on what the function might look like.
I think we should come up with a default implementation that works for any architecture using DT based probing, and requires PCI_DOMAINS to be enabled.
Quoted 5 lines
- alpha, ia64, microblaze, mips, powerpc and tile all define it as a cast of
bus->sysdatato "structpci_controller*" and then access a data member from there to get the domain number. But ... thepci_controllerstructure is different for each architecture, with few more members in addition that might be actually shared with generic framework code.
We do have the generic 'struct pci_host_bridge'. It would be trivial to add a 'domain' field in there and use it in the generic implementation.
Quoted 5 lines
- arm, s390, sparc and x86 have all different names for their sysdata, but they all contain inside a member that is used for giving a domain back. sparc gets an honorary mention here for getting the API wrong and returning
-ENXIOin certain conditions (not like the generic code cares).
I would hope that for the generic case, we can actually remove the use of 'sysdata' and replace it with common code that just looks at pci_host_bridge. Some architectures (s390 in particular) will probably need their own data to be added in sysdata, but overall the architectures are really trying to do the same thing here. With the work that Bjorn and others have done over the past few years, you already need much less architecture specific code. I think it's time to get it to a point where most architectures can do without any at all.
Quoted 11 lines
That takes care of the implementation. But what about usage?
-
drivers/pci/probe.c:pci_create_root_busallocates a new bus structure, sets up the sysdata and ops member pointers and then goes straight into trying to find if the newly created bus doesn't already exist by using the bus number given as parameter andpci_domain_nr()with the new bus structure as argument. I'm trying really hard to figure out what was the intention here, but from the point of view of someone trying to implement thepci_domain_nr()function I have no idea what to return for a virgin bus structure that is not yet attached to any parent.
Right, this needs to be changed when moving the domain into pci_host_bridge.
Quoted 6 lines
So I can see the intent of what Jason is proposing and I'm heading that way myself, but I think I need to cleanup
pci_create_root_busfirst (change the creation order between bridge and bus). And if someone has a good idea on how to determine the domain # from DT we can pluck it into thepcibios_root_bridge_prepare()function (either the generic version or the arch specific one).
How about the change below, to introduce a new pci_scan_domain() function as a variant of pci_scan_root_bus()?
Arnd
drivers/pci/probe.c +48 -3
@@ -1697,8 +1697,9 @@ void __weak pcibios_remove_bus(struct pci_bus *bus) { } -struct pci_bus *pci_create_root_bus (struct device *parent,int bus, -struct pci_ops *ops,void *sysdata,struct list_head *resources) +static struct pci_bus *__pci_create_root_bus (struct device *parent, +int domain,int bus,struct pci_ops *ops,void *sysdata, +struct list_head *resources) {int error;struct pci_host_bridge *bridge;
@@ -1716,7 +1717,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus, b->sysdata = sysdata; b->ops = ops; b->number = b->busn_res.start = bus; - b2 =pci_find_bus (pci_domain_nr (b), bus); + b2 =pci_find_bus (domain, bus);if (b2) {/* If we already got to this bus through a different bridge, ignore it */ dev_dbg (&b2->dev,"bus already known\n" );
@@ -1727,6 +1728,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,if (!bridge) goto err_out; + bridge->domain = domain; bridge->dev.parent = parent; bridge->dev.release = pci_release_host_bridge_dev;dev_set_name (&bridge->dev,"pci%04x:%02x" ,pci_domain_nr (b), bus);
@@ -1801,6 +1803,13 @@ err_out:return NULL ; } +struct pci_bus *pci_create_root_bus (struct device *parent,int bus, +struct pci_ops *ops,void *sysdata,struct list_head *resources) +{ +return __pci_create_root_bus (parent,pci_domain_nr (bus), bus, + ops, sysdata, resources); +} +int pci_bus_insert_busn_res (struct pci_bus *b,int bus,int bus_max) {struct resource *res = &b->busn_res;
@@ -1899,6 +1908,42 @@ struct pci_bus *pci_scan_root_bus(struct device *parent, int bus, }EXPORT_SYMBOL (pci_scan_root_bus); +struct pci_bus *pci_scan_domain (struct device *parent,int domain, +struct pci_ops *ops,void *sysdata,struct list_head *resources) +{ +struct pci_host_bridge_window *window; +bool found = false; +struct pci_bus *b; +int max; + +list_for_each_entry (window, resources, list) +if (window->res->flags &IORESOURCE_BUS ) { + found = true; +break ; + } + +if (!found) { +dev_info (&b->dev, +"No busn resource found for domain %d\n" , domain); +return NULL ; + } + + b =__pci_create_root_bus (parent, domain, window->res->start, + ops, sysdata, resources); +if (!b) +return NULL ; + +dev_info (&b->dev, +"No busn resource found for root bus, will use [bus %02x-ff]\n" , + bus); +pci_bus_insert_busn_res (b, bus,255 ); + } + + max =pci_scan_child_bus (b); + +pci_bus_add_devices (b); +return b; +}/* Deprecated; use pci_scan_root_bus() instead */ struct pci_bus *pci_scan_bus_parented (struct device *parent,int bus,struct pci_ops *ops,void *sysdata)
include/linux/pci.h +1 -0
@@ -395,6 +395,7 @@ struct pci_host_bridge_window {struct pci_host_bridge {struct device dev; +int domain;struct pci_bus *bus;/* root bus */ struct list_head windows;/* pci_host_bridge_windows */ void (*release_fn)(struct pci_host_bridge *);
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 14 lines · Arnd Bergmann
Quoted 13 lines · On Wednesday 19 February 2014 02:44:27 Liviu Dudau wrote:
Quoted 5 lines · Jason Gunthorpe
So the Liviu, I would say the API should be similar to what we see in other OF enabled driver bases subsystems, call the core code with a
platform_devicepointer andfunction_opspointer and have the core code create a domain, figure out the domain # from the DT (via aliases?), and so on.I wish things were easier!
Lets look at the 'int
pci_domain_nr(struct pci_bus *bus);' function. It is used to obtain the domain number of the bus passed as an argument.
Quoted 7 lines
Quoted 3 lines
-
include/linux/pci.hdefines it as an inline function returning zero if !CONFIG_PCI || !CONFIG_PCI_DOMAINS. Otherwise it is silent on what the function might look like.I think we should come up with a default implementation that works for any architecture using DT based probing, and requires
PCI_DOMAINSto be enabled.
Agree. What I was trying to say was that the function is not *defined* if you have CONFIG_PCI && !CONFIG_PCI_DOMAINS, but it has a dummy implementation for !CONFIG_PCI or CONFIG_PCI && CONFIG_PCI_DOMAINS.
Quoted 9 lines
Quoted 6 lines
- alpha, ia64, microblaze, mips, powerpc and tile all define it as a cast of
bus->sysdatato "structpci_controller*" and then access a data member from there to get the domain number. But ... thepci_controllerstructure is different for each architecture, with few more members in addition that might be actually shared with generic framework code.We do have the generic 'struct
pci_host_bridge'. It would be trivial to add a 'domain' field in there and use it in the generic implementation.
That's what my v1 patch was trying to do, but as Tanmay reported, it is not working, see below for the explanation why (short summary, we ask for domain number of a bus before the associated pci_host_bridge is alive).
Quoted 7 lines
Quoted 6 lines
- arm, s390, sparc and x86 have all different names for their sysdata, but they all contain inside a member that is used for giving a domain back. sparc gets an honorary mention here for getting the API wrong and returning
-ENXIOin certain conditions (not like the generic code cares).
Quoted 8 lines
I would hope that for the generic case, we can actually remove the use of 'sysdata' and replace it with common code that just looks at
pci_host_bridge. Some architectures (s390 in particular) will probably need their own data to be added in sysdata, but overall the architectures are really trying to do the same thing here. With the work that Bjorn and others have done over the past few years, you already need much less architecture specific code. I think it's time to get it to a point where most architectures can do without any at all.
We are on the same page. I was just flagging the amount of work we need to do here to bring everyone towards common code.
Quoted 13 lines
Quoted 12 lines
That takes care of the implementation. But what about usage?
-
drivers/pci/probe.c:pci_create_root_busallocates a new bus structure, sets up the sysdata and ops member pointers and then goes straight into trying to find if the newly created bus doesn't already exist by using the bus number given as parameter andpci_domain_nr()with the new bus structure as argument. I'm trying really hard to figure out what was the intention here, but from the point of view of someone trying to implement thepci_domain_nr()function I have no idea what to return for a virgin bus structure that is not yet attached to any parent.
Quoted 1 line
Right, this needs to be changed when moving the domain into
pci_host_bridge.
Yes, if only I understood what the intent was here. Unfortunately, the history of the file stops (for my git tree) at Linus' first commit from v2.6.0, so I need to dig deeper. What I am looking to understand is the answer to this
question: "When you want to create a new bus and you want to make sure it is not duplicated, do you go across all domains or pick the current domain and allow for same bus number in different domains?"
To give some context: I'm looking at drivers/pci/probe.c, pci_create_root_bus(). Specifically, this code:
b = pci_alloc_bus();
if (!b)
return NULL;
b->sysdata = sysdata;
b->ops = ops;
b->number = b->busn_res.start = bus;
b2 = pci_find_bus(pci_domain_nr(b), bus);
if (b2) {
/* If we already got to this bus through a different bridge, ignore it */
dev_dbg(&b2->dev, "bus already known\n");
goto err_out;
}
bridge = pci_alloc_host_bridge(b);
if (!bridge)
goto err_out;If you look at the comment after pci_find_bus, you can see that the intent was to try to find duplicate busses covered by different bridges (but in the same domain? maybe, because there used to be only one domain at the beginning, but then why use pci_domain_nr() here?). Problems I see here:
- code is trying to get the domain number of a bus that it just created but not plugged yet in the hierarchy
- pci_find_bus tries to find a bus in the global list of busses that has the same domain number and bus number as the parameters of the function. But the domain number was plucked out of thin air and all architectures seem to actually give you the current active domain number, not the one that actually covers the bus (and host_bridge) that you are trying to create.
I will try to restructure the code here, but I think the check is superfluous and can be done better by a function that tries to find duplicates when you add the bus, rather than when creating the scaffolding.
Quoted 109 lines
Quoted 7 lines
So I can see the intent of what Jason is proposing and I'm heading that way myself, but I think I need to cleanup
pci_create_root_busfirst (change the creation order between bridge and bus). And if someone has a good idea on how to determine the domain # from DT we can pluck it into thepcibios_root_bridge_prepare()function (either the generic version or the arch specific one).How about the change below, to introduce a new
pci_scan_domain()function as a variant ofpci_scan_root_bus()?Arnd
drivers/pci/probe.c +48
-3@@ -1697,8 +1697,9 @@ void __weak pcibios_remove_bus(struct pci_bus *bus) { } -struct pci_bus *pci_create_root_bus (struct device *parent,int bus, -struct pci_ops *ops,void *sysdata,struct list_head *resources) +static struct pci_bus *__pci_create_root_bus (struct device *parent, +int domain,int bus,struct pci_ops *ops,void *sysdata, +struct list_head *resources) {int error;struct pci_host_bridge *bridge;@@ -1716,7 +1717,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus, b->sysdata = sysdata; b->ops = ops; b->number = b->busn_res.start = bus; - b2 =pci_find_bus (pci_domain_nr (b), bus); + b2 =pci_find_bus (domain, bus);if (b2) {/* If we already got to this bus through a different bridge, ignore it */ dev_dbg (&b2->dev,"bus already known\n" );@@ -1727,6 +1728,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,if (!bridge) goto err_out; + bridge->domain = domain; bridge->dev.parent = parent; bridge->dev.release = pci_release_host_bridge_dev;dev_set_name (&bridge->dev,"pci%04x:%02x" ,pci_domain_nr (b), bus);@@ -1801,6 +1803,13 @@ err_out:return NULL ; } +struct pci_bus *pci_create_root_bus (struct device *parent,int bus, +struct pci_ops *ops,void *sysdata,struct list_head *resources) +{ +return __pci_create_root_bus (parent,pci_domain_nr (bus), bus, + ops, sysdata, resources); +} +int pci_bus_insert_busn_res (struct pci_bus *b,int bus,int bus_max) {struct resource *res = &b->busn_res;@@ -1899,6 +1908,42 @@ struct pci_bus *pci_scan_root_bus(struct device *parent, int bus, }EXPORT_SYMBOL (pci_scan_root_bus); +struct pci_bus *pci_scan_domain (struct device *parent,int domain, +struct pci_ops *ops,void *sysdata,struct list_head *resources) +{ +struct pci_host_bridge_window *window; +bool found = false; +struct pci_bus *b; +int max; + +list_for_each_entry (window, resources, list) +if (window->res->flags &IORESOURCE_BUS ) { + found = true; +break ; + } + +if (!found) { +dev_info (&b->dev, +"No busn resource found for domain %d\n" , domain); +return NULL ; + } + + b =__pci_create_root_bus (parent, domain, window->res->start, + ops, sysdata, resources); +if (!b) +return NULL ; + +dev_info (&b->dev, +"No busn resource found for root bus, will use [bus %02x-ff]\n" , + bus); +pci_bus_insert_busn_res (b, bus,255 ); + } + + max =pci_scan_child_bus (b); + +pci_bus_add_devices (b); +return b; +}/* Deprecated; use pci_scan_root_bus() instead */ struct pci_bus *pci_scan_bus_parented (struct device *parent,int bus,struct pci_ops *ops,void *sysdata)include/linux/pci.h +0
-0
include/linux/pci.h struct pci_host_bridge_window { · line 401
Quoted 7 lines
include/linux/pci.h +1
-0@@ -395,6 +395,7 @@ struct pci_host_bridge_window {struct pci_host_bridge {struct device dev; +int domain;struct pci_bus *bus;/* root bus */ struct list_head windows;/* pci_host_bridge_windows */ void (*release_fn)(struct pci_host_bridge *);
Yes, I think this will work, as long as we can figure out if the intent of the code is corect.
Thanks,
Liviu
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 9 lines · On Wednesday 19 February 2014 11:37:55 Liviu Dudau wrote:
Quoted 2 lines
Right, this needs to be changed when moving the domain into
pci_host_bridge.Yes, if only I understood what the intent was here. Unfortunately, the history of the file stops (for my git tree) at Linus' first commit from v2.6.0, so I need to dig deeper. What I am looking to understand is the answer to this
question: "When you want to create a new bus and you want to make sure it is not duplicated, do you go across all domains or pick the current domain and allow for same bus number in different domains?"
I suspect this is for some large IA64 machines where you can see the same domain at different addresses, depending on which CPU you are on, or something similarly fancy.
Another explanation would be that you can have different ways to describe the same hardware: On a PC you can assume that the PCI bus is discoverable on I/O port 0x0cf8. At the same time, you may see it in ACPI with an mmconfig method.
In any case, you definitely need to allow the same bus numbers on each domain, since the most important use case of domains is to get around the 256 bus limit.
Quoted 22 lines
To give some context: I'm looking at
drivers/pci/probe.c,pci_create_root_bus(). Specifically, this code:b = pci_alloc_bus(); if (!b) return NULL; b->sysdata = sysdata; b->ops = ops; b->number = b->busn_res.start = bus; b2 = pci_find_bus(pci_domain_nr(b), bus); if (b2) { /* If we already got to this bus through a different bridge, ignore it */ dev_dbg(&b2->dev, "bus already known\n"); goto err_out; } bridge = pci_alloc_host_bridge(b); if (!bridge) goto err_out;
Quoted 4 lines
If you look at the comment after
pci_find_bus, you can see that the intent was to try to find duplicate busses covered by different bridges (but in the same domain? maybe, because there used to be only one domain at the beginning, but then why usepci_domain_nr()here?). Problems I see here:
Definitely duplicate buses in the same domain, as mentioned above.
Quoted 2 lines
- code is trying to get the domain number of a bus that it just created but not plugged yet in the hierarchy
The sysdata is already assigned, and all architectures either assume there is only one domain, or they get the domain number out of the sysdata that is filled out prior to calling pci_create_root_bus.
Quoted 5 lines
-
pci_find_bustries to find a bus in the global list of busses that has the same domain number and bus number as the parameters of the function. But the domain number was plucked out of thin air and all architectures seem to actually give you the current active domain number, not the one that actually covers the bus (andhost_bridge) that you are trying to create.
Not sure what you mean with 'current active domain number': They clearly look at b->sysdata here, which is a local structure describing what we are currently probing.
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 51 lines · Arnd Bergmann
Quoted 9 lines · On Wednesday 19 February 2014 11:37:55 Liviu Dudau wrote:
Quoted 2 lines
Right, this needs to be changed when moving the domain into
pci_host_bridge.Yes, if only I understood what the intent was here. Unfortunately, the history of the file stops (for my git tree) at Linus' first commit from v2.6.0, so I need to dig deeper. What I am looking to understand is the answer to this
question: "When you want to create a new bus and you want to make sure it is not duplicated, do you go across all domains or pick the current domain and allow for same bus number in different domains?"I suspect this is for some large IA64 machines where you can see the same domain at different addresses, depending on which CPU you are on, or something similarly fancy.
Another explanation would be that you can have different ways to describe the same hardware: On a PC you can assume that the PCI bus is discoverable on I/O port
0x0cf8. At the same time, you may see it in ACPI with an mmconfig method.In any case, you definitely need to allow the same bus numbers on each domain, since the most important use case of domains is to get around the 256 bus limit.
Quoted 22 lines
To give some context: I'm looking at
drivers/pci/probe.c,pci_create_root_bus(). Specifically, this code:b = pci_alloc_bus(); if (!b) return NULL; b->sysdata = sysdata; b->ops = ops; b->number = b->busn_res.start = bus; b2 = pci_find_bus(pci_domain_nr(b), bus); if (b2) { /* If we already got to this bus through a different bridge, ignore it */ dev_dbg(&b2->dev, "bus already known\n"); goto err_out; } bridge = pci_alloc_host_bridge(b); if (!bridge) goto err_out;Quoted 4 lines
If you look at the comment after
pci_find_bus, you can see that the intent was to try to find duplicate busses covered by different bridges (but in the same domain? maybe, because there used to be only one domain at the beginning, but then why usepci_domain_nr()here?). Problems I see here:
Quoted 8 lines
Definitely duplicate buses in the same domain, as mentioned above.
Quoted 2 lines
- code is trying to get the domain number of a bus that it just created but not plugged yet in the hierarchy
The sysdata is already assigned, and all architectures either assume there is only one domain, or they get the domain number out of the sysdata that is filled out prior to calling
pci_create_root_bus.
Arnd, the question is not how it does it, the question is what is the correct answer? It's a new bus, does that mean that all new busses get created in the current domain? If that is the case, then fine, things start to make more sense, with the caveat that the domain # depends heavily on what the architecture uses for counting.
Quoted 10 lines
Quoted 6 lines
-
pci_find_bustries to find a bus in the global list of busses that has the same domain number and bus number as the parameters of the function. But the domain number was plucked out of thin air and all architectures seem to actually give you the current active domain number, not the one that actually covers the bus (andhost_bridge) that you are trying to create.Not sure what you mean with 'current active domain number': They clearly look at
b->sysdatahere, which is a local structure describing what we are currently probing.
Yes, but that definition of locality is not common to all architectures. arm for example stores it in the domain member of pci_sys_data, but that gets initialised every time someone calls pcibios_init_hw (or pci_common_init_dev) with the domain value held in struct hw_pci. But the host controllers don't talk to each other, so if you want to instantiate two different drivers that use hw_pci they likely use an individual count that starts at zero.
Anyway, I don't think that using your suggestion is going to be a problem, so once I get out of this god forsaken task of bringing up a board I will post an updated patch.
Best regards,
Liviu
Quoted 3 lines
Arnd
Signature · 10 lines
-------------------
.oooO
( )
\ ( Oooo.
\_) ( )
) /
(_/
One small step
for me ...PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 6 lines · On Wednesday 19 February 2014 15:30:44 Liviu Dudau wrote:
Quoted 5 lines · Arnd Bergmann
Quoted 2 lines · On Wednesday 19 February 2014 11:37:55 Liviu Dudau wrote:
- code is trying to get the domain number of a bus that it just created but not plugged yet in the hierarchy
Quoted 8 lines
Quoted 3 lines
The sysdata is already assigned, and all architectures either assume there is only one domain, or they get the domain number out of the sysdata that is filled out prior to calling
pci_create_root_bus.Arnd, the question is not how it does it, the question is what is the correct answer? It's a new bus, does that mean that all new busses get created in the current domain? If that is the case, then fine, things start to make more sense, with the caveat that the domain # depends heavily on what the architecture uses for counting.
Here is what each architecture does:
Alpha: managed by host bridge driver, counting the hosts or fixed in hardware
ARM: managed in host bridge driver, counting the hosts or using just one domain
IA64: taken from firmware (either ACPI or SAL)
microblaze: managed globally, counting the hosts
powerpc: managed globally, counting the hosts
s390: each PCI function has its own domain (!)
sh: managed in host bridge driver, counting the hosts
sparc: managed by host bridge driger, counting the hosts
tile: managed globally, counting the hosts
x86: managed by ACPI firmware
In each case, we either know the domain from hardware or firmware when creating the host bridge, or we pick the next free number.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
-
pci_find_bustries to find a bus in the global list of busses that has the same domain number and bus number as the parameters of the function. But the domain number was plucked out of thin air and all architectures seem to actually give you the current active domain number, not the one that actually covers the bus (andhost_bridge) that you are trying to create.Not sure what you mean with 'current active domain number': They clearly look at
b->sysdatahere, which is a local structure describing what we are currently probing.
Quoted 6 lines
Yes, but that definition of locality is not common to all architectures. arm for example stores it in the domain member of
pci_sys_data, but that gets initialised every time someone callspcibios_init_hw(orpci_common_init_dev) with the domain value held in structhw_pci. But the host controllers don't talk to each other, so if you want to instantiate two different drivers that usehw_pcithey likely use an individual count that starts at zero.
The assumption is that whoever calls pci_common_init{_dev} knows about all the PCI hosts in the system. As Jason Gunthorpe said, it would be desirable to have stable domain numbers, so it's probably best to take the number from DT (e.g. the aliases node). It's probably safe to assume that any machine using ATAGS for booting falls into the category where it's safe to keep the policy local in the host bridge driver.
Arnd
Bjorn Helgaas
maintainerPATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Arnd BergmannQuoted 40 lines · Arnd Bergmann
Quoted 8 lines · On Friday 14 February 2014 22:00:47 Liviu Dudau wrote:
What I'm going to suggest in my v2 patch (hope to send it before Monday) is a new API in the generic PCI code that will allow you to create a host bridge in a new domain or in the existing domain, with the management of the domain number being done in the generic code.
Something like:
Quoted 7 lines
int create_hostbridge_in_new_domain(....); int create_hostbridge(....);with the functions being wrappers around the
pci_hostbridge_of_initfunction that I'm introducing.What do you think?
Not sure. It would still keep the decision about whether to use a new domain or not in the host bridge driver, but that doesn't seem like the right place. The same driver might be used in different ways based on what is around it.
I've also had a look at the MIPS implementation now, which has its own way of dealing with this, see
arch/mips/pci/pci.c.There was also an interesting comment in the MIPS header file:
/* For compatibility with current (as of July 2003) pciutils and XFree86. Eventually will be removed. */ unsigned int need_domain_info;This is over ten years old, so I wonder if we can start assuming that domains work out of the box now. All references to problems from PCI domains are about old code (ca. pre-2007) that doesn't understand nonzero domains and that has since been fixed. I am pretty sure we don't need to ever worry about stuffing multiple host bridges into a domain other than the first one, and I also doubt we have to worry about the problem at all on arm64 as we won't run old binaries on it (or arm32 compat mode binaries that need to manage PCI devices).
Quoted 3 lines
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
With ACPI on x86 and ia64, we currently use _SEG directly as the Linux PCI domain. Are you proposing that we make the Linux PCI domain independent of _SEG?
It will look sort of funny to have things like this:
ACPI: PCI Root Bridge [L000] (_SEG 0 domain 0000 [bus 00-1b])
ACPI: PCI Root Bridge [L001] (_SEG 0 domain 0001 [bus 1c-37])
ACPI: PCI Root Bridge [L002] (_SEG 0 domain 0002 [bus 38-53])
where the firmware had _SEG==0 for all the bridges and assigned non-overlapping bus number ranges, but since nothing in PCI really depends on the domain, I guess it should work.
Bjorn
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 10 lines · On Tuesday 18 February 2014 17:28:14 Bjorn Helgaas wrote:
Quoted 5 lines · Arnd Bergmann
On Friday 14 February 2014 22:00:47 Liviu Dudau wrote:
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
With ACPI on x86 and ia64, we currently use
_SEGdirectly as the Linux PCI domain. Are you proposing that we make the Linux PCI domain independent of_SEG?
I don't think we should change anything for ACPI.
The point is that we don't currently have the equivalent of _SEG for DT probing. The best two options we have are:
a) introduce a 'pci-segment' property with the same meaning as _SEG on ACPI, and use that as the domain number
b) don't introduce the segment concept on DT but just give each host bridge its own domain.
The second one seems a little easier to implement, and I don't see what _SEG is used for other than to avoid having domains when you don't need them. Is there more to it that I'm missing?
Quoted 9 lines
It will look sort of funny to have things like this:
ACPI: PCI Root Bridge [L000] (
_SEG0 domain 0000 [bus 00-1b])
ACPI: PCI Root Bridge [L001] (_SEG0 domain 0001 [bus 1c-37])
ACPI: PCI Root Bridge [L002] (_SEG0 domain 0002 [bus 38-53])where the firmware had _SEG==0 for all the bridges and assigned non-overlapping bus number ranges, but since nothing in PCI really depends on the domain, I guess it should work.
I would expect that with DT, this would look like
Root bridge 0: domain 0 [bus 0-1b]
Root bridge 1: domain 1 [bus 0-1b]
Root bridge 2: domain 2 [bus 0-1b]
Since you wouldn't have a reason to use a bus range starting at a nonzero number.
Arnd
Bjorn Helgaas
maintainerPATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 19 lines · Arnd Bergmann
Quoted 10 lines · On Tuesday 18 February 2014 17:28:14 Bjorn Helgaas wrote:
Quoted 5 lines · Arnd Bergmann
On Friday 14 February 2014 22:00:47 Liviu Dudau wrote:
Can anyone with more experience on the subject than me (Bjorn, Russell, Jason, ...) think of a reason why we would not want to just use a new domain for every host bridge we find?
With ACPI on x86 and ia64, we currently use
_SEGdirectly as the Linux PCI domain. Are you proposing that we make the Linux PCI domain independent of_SEG?I don't think we should change anything for ACPI.
The point is that we don't currently have the equivalent of_SEGfor DT probing. The best two options we have are:a) introduce a 'pci-segment' property with the same meaning as
_SEGon ACPI, and use that as the domain number
Quoted 6 lines
b) don't introduce the segment concept on DT but just give each host bridge its own domain.
The second one seems a little easier to implement, and I don't see what
_SEGis used for other than to avoid having domains when you don't need them. Is there more to it that I'm missing?
Not really, but I do have a question related to OS management of host bridge bus number apertures. Currently, Linux never changes a host bridge's bus number range, but it's conceivable that we could in some hotplug scenario. ACPI does provide a mechanism to do so (_PRS, _SRS), and other host bridge drivers could also do this by programming CSRs to change the bus number range. The PCI domain is the logical place to manage allocation of the 00-ff range of bus numbers.
1) x86 platforms may have constraints because PCIBIOS and the 0xcf8 config access mechanism are unaware of segments. If a platform has to support legacy OSes that use those, it can't reuse bus numbers even in different segment groups. The platform might have to use multiple segments to allow multiple ECAM regions, but use _PRS/_SRS to prevent bus number overlaps to keep legacy config access working. Obviously this is only an issue if there are non-segment aware config access methods.
2) If two host bridges share an ECAM region, I think we're forced to put them in the same domain: if we put them in different domains, Linux might assign [bus 00-ff] to both bridges, and ECAM config accesses would only work for one of the bridges. This is quite common on x86 and is a potential issue for any architecture.
Bjorn
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 15 lines · On Wednesday 19 February 2014 11:20:19 Bjorn Helgaas wrote:
Quoted 5 lines · Arnd Bergmann
On Tuesday 18 February 2014 17:28:14 Bjorn Helgaas wrote:
The second one seems a little easier to implement, and I don't see what
_SEGis used for other than to avoid having domains when you don't need them. Is there more to it that I'm missing?Not really, but I do have a question related to OS management of host bridge bus number apertures. Currently, Linux never changes a host bridge's bus number range, but it's conceivable that we could in some hotplug scenario. ACPI does provide a mechanism to do so (
_PRS,_SRS), and other host bridge drivers could also do this by programming CSRs to change the bus number range. The PCI domain is the logical place to manage allocation of the 00-ff range of bus numbers.
Quoted 8 lines
1) x86 platforms may have constraints because PCIBIOS and the
0xcf8config access mechanism are unaware of segments. If a platform has to support legacy OSes that use those, it can't reuse bus numbers even in different segment groups. The platform might have to use multiple segments to allow multiple ECAM regions, but use _PRS/_SRS to prevent bus number overlaps to keep legacy config access working. Obviously this is only an issue if there are non-segment aware config access methods.
Right, I don't think this will be an issue outside of x86/ia64/alpha, since on all other architectures I'm aware of you have no PCIBIOS and each host controller would also have its own config space. Even host controllers using 0xfc8 would be fine because each host bridge normally has its own I/O space.
Quoted 5 lines
2) If two host bridges share an ECAM region, I think we're forced to put them in the same domain: if we put them in different domains, Linux might assign [bus 00-ff] to both bridges, and ECAM config accesses would only work for one of the bridges. This is quite common on x86 and is a potential issue for any architecture.
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.
I still want to make sure I understand exactly what this case is about though, i.e. what is shared and what is separate if you have two host bridges with a common ECAM region:
* I assume I/O space is always shared on x86, but probably separate elsewhere.
* Each host would always have a fixed memory space aperture, right?
* From what I understand from your description, the hardware does not enforce specific bus numbers for each host. How does the host bridge know its root bus number then?
* Should I expect one IOMMU per host bridge or one ECAM region, or can either be possible?
* The IntA-IntB IRQ numbers are always per host bridge I assume.
* Memory space on one host bridge is visible to bus master DMA from a device on another host bridge on x86, right? I assume this won't normally be the case on other architectures.
Arnd
Bjorn Helgaas
maintainerPATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 37 lines · Arnd Bergmann
Quoted 15 lines · On Wednesday 19 February 2014 11:20:19 Bjorn Helgaas wrote:
Quoted 5 lines · Arnd Bergmann
On Tuesday 18 February 2014 17:28:14 Bjorn Helgaas wrote:
The second one seems a little easier to implement, and I don't see what
_SEGis used for other than to avoid having domains when you don't need them. Is there more to it that I'm missing?Not really, but I do have a question related to OS management of host bridge bus number apertures. Currently, Linux never changes a host bridge's bus number range, but it's conceivable that we could in some hotplug scenario. ACPI does provide a mechanism to do so (
_PRS,_SRS), and other host bridge drivers could also do this by programming CSRs to change the bus number range. The PCI domain is the logical place to manage allocation of the 00-ff range of bus numbers.Quoted 8 lines
1) x86 platforms may have constraints because PCIBIOS and the
0xcf8config access mechanism are unaware of segments. If a platform has to support legacy OSes that use those, it can't reuse bus numbers even in different segment groups. The platform might have to use multiple segments to allow multiple ECAM regions, but use _PRS/_SRS to prevent bus number overlaps to keep legacy config access working. Obviously this is only an issue if there are non-segment aware config access methods.Right, I don't think this will be an issue outside of x86/ia64/alpha, since on all other architectures I'm aware of you have no PCIBIOS and each host controller would also have its own config space. Even host controllers using
0xfc8would be fine because each host bridge normally has its own I/O space.Quoted 5 lines
2) If two host bridges share an ECAM region, I think we're forced to put them in the same domain: if we put them in different domains, Linux might assign [bus 00-ff] to both bridges, and ECAM config accesses would only work for one of the bridges. This is quite common on x86 and is a potential issue for any architecture.
Quoted 11 lines
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.
Addresses in the ACPI MCFG table correspond to bus number 0, but the MCFG also provides start & end bus numbers, so the valid range does not necessarily start with bus 0 and need not be power-of-two in size.
Something similar sounds like a good idea for DT.
Quoted 6 lines
I still want to make sure I understand exactly what this case is about though, i.e. what is shared and what is separate if you have two host bridges with a common ECAM region:
* I assume I/O space is always shared on x86, but probably separate elsewhere.
I think x86 *could* support multiple I/O spaces, but user-mode inb/outb could only reach the 0-64K space. I think ia64 has the same limitation on user code, but it supports many spaces in the kernel.
Quoted 1 line
* Each host would always have a fixed memory space aperture, right?
The ACPI _CRS/_PRS/_SRS mechanism theoretically allows changes to the bus number, I/O space, and memory space apertures of host bridges. But we don't do any of those changes today, and I don't know if any BIOSes actually allow it.
Quoted 3 lines
* From what I understand from your description, the hardware does not enforce specific bus numbers for each host. How does the host bridge know its root bus number then?
I don't know details of any specific hardware. I'm just saying that ACPI provides a mechanism for the OS to manipulate the bus number range below a host bridge. Of course, a BIOS is free to omit _PRS and _SRS, and in that case, the bus/IO/memory apertures reported by _CRS are fixed and can't be changed. We learn the root bus number from the host bridge _CRS (but I'm sure you knew that, so maybe I missed the point of your question).
Quoted 2 lines
* Should I expect one IOMMU per host bridge or one ECAM region, or can either be possible?
It's possible to have multiple IOMMUs per host bridge, and I think they can even be buried down in the PCIe hierarchy.
Quoted 1 line
* The IntA-IntB IRQ numbers are always per host bridge I assume.
For conventional PCI, INTx are just wires that could go anywhere, so there's no connection between them and a host bridge. You have to have a _PRT or similar to make sense out of them. For PCIe, a Root Complex maps INTx emulation messages to system interrupts in an implementation-specific way, so we need a _PRT there, too. I don't think there's a requirement that these IRQ numbers be per-host bridge.
Quoted 3 lines
* Memory space on one host bridge is visible to bus master DMA from a device on another host bridge on x86, right? I assume this won't normally be the case on other architectures.
I think this is also implementation dependent, and I'm not aware of an ACPI or other generic way to learn what a particular platform does. It seems like this was an issue for MPS configuration, where peer-to-peer DMA is a problem because we don't really know how Root Complexes handle peer-to-peer transactions.
Bjorn
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 13 lines · On Wednesday 19 February 2014 13:18:24 Bjorn Helgaas wrote:
Quoted 12 lines
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.
Quoted 4 lines
Addresses in the ACPI MCFG table correspond to bus number 0, but the MCFG also provides start & end bus numbers, so the valid range does not necessarily start with bus 0 and need not be power-of-two in size.
Something similar sounds like a good idea for DT.
Hmm, we'll have to think about that. From a DT perspective, we try to keep things local to the node using it, so listing only the registers we are allowed to access is more natural.
Another option would be to have a separate device node for the ECAM registers and point to that from the host controller node, which would describe this cleanly but also add a bit of complexity that will rarely be used.
Quoted 10 lines
Quoted 6 lines
I still want to make sure I understand exactly what this case is about though, i.e. what is shared and what is separate if you have two host bridges with a common ECAM region:
* I assume I/O space is always shared on x86, but probably separate elsewhere.
I think x86 *could* support multiple I/O spaces, but user-mode inb/outb could only reach the 0-64K space. I think ia64 has the same limitation on user code, but it supports many spaces in the kernel.
Ok.
Quoted 6 lines
Quoted 1 line
* Each host would always have a fixed memory space aperture, right?
The ACPI _CRS/_PRS/_SRS mechanism theoretically allows changes to the bus number, I/O space, and memory space apertures of host bridges. But we don't do any of those changes today, and I don't know if any BIOSes actually allow it.
I mean non-overlapping apertures in particular. We also have cases where the aperture that we list in DT is just programmed into hardware registers by the host driver and could be arbitrary, but you can't normally have the same MMIO address go to two devices on internal buses (or not in a sensible way).
Quoted 11 lines
Quoted 3 lines
* From what I understand from your description, the hardware does not enforce specific bus numbers for each host. How does the host bridge know its root bus number then?
I don't know details of any specific hardware. I'm just saying that ACPI provides a mechanism for the OS to manipulate the bus number range below a host bridge. Of course, a BIOS is free to omit
_PRSand_SRS, and in that case, the bus/IO/memory apertures reported by_CRSare fixed and can't be changed. We learn the root bus number from the host bridge_CRS(but I'm sure you knew that, so maybe I missed the point of your question).
I guess the answer then is that the host bridge can have a register for programming the root bus number, but it's not standardized and therefore the access is hidden in the _PRS/_SRS methods. If we have the same on DT and want to reprogram the bus numbers, we'd need to have a kernel driver for the nonstandard registers of the host bridge.
Quoted 5 lines
Quoted 2 lines
* Should I expect one IOMMU per host bridge or one ECAM region, or can either be possible?
It's possible to have multiple IOMMUs per host bridge, and I think they can even be buried down in the PCIe hierarchy.
Oh, I didn't know that. So how do you actually find the IOMMU for a given domain/bus/device/function combination?
Quoted 8 lines
Quoted 1 line
* The IntA-IntB IRQ numbers are always per host bridge I assume.
For conventional PCI, INTx are just wires that could go anywhere, so there's no connection between them and a host bridge. You have to have a
_PRTor similar to make sense out of them. For PCIe, a Root Complex maps INTx emulation messages to system interrupts in an implementation-specific way, so we need a_PRTthere, too. I don't think there's a requirement that these IRQ numbers be per-host bridge.
Right, makes sense.
Thanks for the detailed explanations!
Arnd
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 3 lines · Arnd Bergmann
Hmm, we'll have to think about that. From a DT perspective, we try to keep things local to the node using it, so listing only the registers we are allowed to access is more natural.
If I understand the restriction properly, in a modern PCI-E world it boils down to a limition on the configuration of each PCI-PCI root port bridge (eg, a limited range of valid bus values, and apertures)
AFAIK it comes from the hidden per-socket routing registers that the firwmare configures. Range X->Y (bus #, IO and MMIO) will be routed to a specific physical socket, and then the PCI-E bridges in that socket claim the transatcion based on their local config to select the ultimate egress port.
So describing and restricting the bridge DT node itself, under a single top level PCI domain stanza seems pretty reasonable.
As does containing the restrictions in a HW driver with knowledge of the hidden registers, especially for firmware-less embedded. This is part of what mvebu is doing already.
Jason
Bjorn Helgaas
maintainerPATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Arnd BergmannQuoted 14 lines · Arnd Bergmann
Quoted 13 lines · On Wednesday 19 February 2014 13:18:24 Bjorn Helgaas wrote:
Quoted 12 lines
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.
Quoted 8 lines
Quoted 4 lines
Addresses in the ACPI MCFG table correspond to bus number 0, but the MCFG also provides start & end bus numbers, so the valid range does not necessarily start with bus 0 and need not be power-of-two in size.
Something similar sounds like a good idea for DT.Hmm, we'll have to think about that. From a DT perspective, we try to keep things local to the node using it, so listing only the registers we are allowed to access is more natural.
The combination of MCFG base address for bus 00 and the bus number range from _CRS, plus the obvious offset computation does effectively describe the registers you're allowed to access; it's just up to the OS to compute the offsets. _CBA (an optional method that returns the ECAM address for a hot-added host bridge) uses the same bus 00 base. My guess is that _CBA uses a bus number 00 base so it can return a constant, regardless of whether the OS changes the bus number range. If _CBA returned the ECAM base for the current bus number aperture, it would be dependent on _CRS (the current settings), and the OS would have to re-evaluate _CBA if it ever changed the bus number aperture.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
* Each host would always have a fixed memory space aperture, right?
The ACPI _CRS/_PRS/_SRS mechanism theoretically allows changes to the bus number, I/O space, and memory space apertures of host bridges. But we don't do any of those changes today, and I don't know if any BIOSes actually allow it.
Quoted 5 lines
I mean non-overlapping apertures in particular. We also have cases where the aperture that we list in DT is just programmed into hardware registers by the host driver and could be arbitrary, but you can't normally have the same MMIO address go to two devices on internal buses (or not in a sensible way).
I don't remember specific spec statements about that, but I can't imagine how to make sense of an address that's claimed by two devices.
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
* From what I understand from your description, the hardware does not enforce specific bus numbers for each host. How does the host bridge know its root bus number then?
I don't know details of any specific hardware. I'm just saying that ACPI provides a mechanism for the OS to manipulate the bus number range below a host bridge. Of course, a BIOS is free to omit
_PRSand_SRS, and in that case, the bus/IO/memory apertures reported by_CRSare fixed and can't be changed. We learn the root bus number from the host bridge_CRS(but I'm sure you knew that, so maybe I missed the point of your question).
Quoted 5 lines
I guess the answer then is that the host bridge can have a register for programming the root bus number, but it's not standardized and therefore the access is hidden in the _PRS/_SRS methods. If we have the same on DT and want to reprogram the bus numbers, we'd need to have a kernel driver for the nonstandard registers of the host bridge.
Exactly; that's my mental model of how it works: _CRS/_PRS/_SRS are basically accessors for generalized BARs.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
* Should I expect one IOMMU per host bridge or one ECAM region, or can either be possible?
It's possible to have multiple IOMMUs per host bridge, and I think they can even be buried down in the PCIe hierarchy.
Oh, I didn't know that. So how do you actually find the IOMMU for a given domain/bus/device/function combination?
For VT-d on x86, there's a DMAR table that describes the remapping units (IOMMUs), and each has a list of associated devices. This is one place where the FW/OS interface uses segment and bus numbers. There's something different for AMD IOMMUs, but I think it also involves looking up the device in a table from the firmware.
Bjorn
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 20 lines · On Wednesday 19 February 2014 14:33:54 Bjorn Helgaas wrote:
Quoted 19 lines · Arnd Bergmann
Quoted 17 lines · On Wednesday 19 February 2014 13:18:24 Bjorn Helgaas wrote:
Quoted 12 lines
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.Addresses in the ACPI MCFG table correspond to bus number 0, but the MCFG also provides start & end bus numbers, so the valid range does not necessarily start with bus 0 and need not be power-of-two in size.
Something similar sounds like a good idea for DT.
Quoted 8 lines
Quoted 3 lines
Hmm, we'll have to think about that. From a DT perspective, we try to keep things local to the node using it, so listing only the registers we are allowed to access is more natural.
The combination of MCFG base address for bus 00 and the bus number range from
_CRS, plus the obvious offset computation does effectively describe the registers you're allowed to access; it's just up to the OS to compute the offsets.
That's not what I meant. The 'reg' property is supposed to list the registers that are exclusively owned by the device. We normally expect that we want to request_mem_region() and ioremap() the entire range for each device, which doesn't make sense if we only want a small part of it, or if another device owns the same registers.
Having a separate device that just owns the ECAM region would work fine, because then it can ioremap that once and get referenced by the host bridge drivers.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
* Should I expect one IOMMU per host bridge or one ECAM region, or can either be possible?
It's possible to have multiple IOMMUs per host bridge, and I think they can even be buried down in the PCIe hierarchy.
Quoted 8 lines
Quoted 2 lines
Oh, I didn't know that. So how do you actually find the IOMMU for a given domain/bus/device/function combination?
For VT-d on x86, there's a DMAR table that describes the remapping units (IOMMUs), and each has a list of associated devices. This is one place where the FW/OS interface uses segment and bus numbers. There's something different for AMD IOMMUs, but I think it also involves looking up the device in a table from the firmware.
Ok, that seems complicated. I hope we don't have to get this deep on ARM and can keep it on a per host-bridge basis rather than having to get down all the way to the device. We wouldn't be able to encode IOMMUs for hotplugged devices in DT, because the firmware is no longer running by the time they show up.
Arnd
Bjorn Helgaas
maintainerPATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 30 lines · Arnd Bergmann
Quoted 20 lines · On Wednesday 19 February 2014 14:33:54 Bjorn Helgaas wrote:
Quoted 19 lines · Arnd Bergmann
Quoted 17 lines · On Wednesday 19 February 2014 13:18:24 Bjorn Helgaas wrote:
Quoted 12 lines
Right, this is an interesting case indeed, and I think we haven't considered it in the binding so far. We already encode a "bus-range" in DT, so we can easily partition the ECAM config space, but it still violates one of the two assumptions:
a) that the register ranges for the two host bridge devices are non-overlapping in DT
b) that the ECAM register range as specified in DT starts at bus 0 and is a power-of-two size.
Since the binding is not fixed that, we could change the definition to say that the ECAM register range in the "reg" property must match the buses listed in the "bus-range" property.Addresses in the ACPI MCFG table correspond to bus number 0, but the MCFG also provides start & end bus numbers, so the valid range does not necessarily start with bus 0 and need not be power-of-two in size.
Something similar sounds like a good idea for DT.Quoted 8 lines
Quoted 3 lines
Hmm, we'll have to think about that. From a DT perspective, we try to keep things local to the node using it, so listing only the registers we are allowed to access is more natural.
The combination of MCFG base address for bus 00 and the bus number range from
_CRS, plus the obvious offset computation does effectively describe the registers you're allowed to access; it's just up to the OS to compute the offsets.
Quoted 5 lines
That's not what I meant. The 'reg' property is supposed to list the registers that are exclusively owned by the device. We normally expect that we want to
request_mem_region()andioremap()the entire range for each device, which doesn't make sense if we only want a small part of it, or if another device owns the same registers.
Oh, right, that makes good sense. The ACPI approach means we have to have special-case code for that; it can't be handled by request_mem_region() in a generic _CRS parser.
Bjorn
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Kumar GalaQuoted 53 lines · Kumar Gala
Quoted 50 lines · Will Deacon
Quoted 48 lines · Kumar Gala
Quoted 32 lines · Will Deacon
This patch adds support for a generic PCI host controller, such as a firmware-initialised device with static windows or an emulation by something such as kvmtool.
The controller itself has no configuration registers and has its address spaces described entirely by the device-tree (using the bindings from ePAPR). Both CAM and ECAM are supported for Config Space accesses.
Corresponding documentation is added for the DT binding.
Signed-off-by Will Deacon <[email protected]>---
.../devicetree/bindings/pci/arm-generic-pci.txt | 51 ++++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-arm-generic.c | 318 +++++++++++++++++++++ 4 files changed, 377 insertions(+)create mode 100644
Documentation/devicetree/bindings/pci/arm-generic-pci.txtcreate mode 100644drivers/pci/host/pci-arm-generic.cDocumentation/devicetree/bindings/pci/arm-generic-pci.txt new file +6
-0@@ -0,0 +1,51 @@ +* ARM generic PCI host controller + +Firmware-initialised PCI host controllers and PCI emulations, such as the +virtio-pci implementations found in kvmtool and other para-virtualised +systems, do not require driver support for complexities such as regulator and +clock management. In fact, the controller may not even require theDocumentation/devicetree/bindings/pci/arm-generic-pci.txt line 16
Quoted 10 lines
Documentation/devicetree/bindings/pci/arm-generic-pci.txt new file +10
-0+configuration of a control interface by the operating system, instead +presenting a set of fixed windows describing a subset of IO, Memory and +Configuration Spaces. + +Such a controller can be described purely in terms of the standardized device +tree bindings communicated in pci.txt: + +- compatible : Must be "arm,pci-cam-generic" or "arm,pci-ecam-generic" + depending on the layout of configuration space (CAM vs + ECAM respectively)What's arm specific here? I don't have a great suggestion, but seems odd for this to be vendor prefixed with "arm".
Quoted 3 lines
Quoted 1 line
Happy to change it, but I'm also struggling for names. Maybe "linux,..."?
I was thinking that as well, I'd say go with "linux,".
Just drop the prefix altogether.
I'm wondering if this should have host or rc in the name.
Rob
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
In reply to Will DeaconQuoted 1 line · Will Deacon
Not in this case! kvmtool generates the following:
Well, the example is nice so someone can review it..
Quoted 10 lines
pci {
#address-cells = <0x3>;
#size-cells = <0x2>;
#interrupt-cells = <0x1>;
compatible = "arm,pci-cam-generic";
reg = <0x0 0x40000000>;
ranges = <0x1000000 0x0 0x0 0x0 0x0 0x0 0x10000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>;
interrupt-map = <0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1 0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1 0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1 0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>;
interrupt-map-mask = <0xf800 0x0 0x0 0x7>;
};Looks like there are a few misses in the above. How about this:
pci {
compatible = "arm,pci-cam-generic"
device_type = "pci";
** ^^^^^^^^^^^^^^^^^
** MANDATORY for the host bridge node
#address-cells = <3>;
#size-cells = <2>;
// BUS_ADDRESS(3) CPU_PHYSICAL(2) SIZE(2)
ranges = <0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000>,
^^^^^^^^^^^^^^
** ?? Is this why you had problems with the offset? Should
** be the cpu physical address of the start of the IO window.
<0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>;
#interrupt-cells = <0x1>;
// PCI_DEVICE(3) INT#(1) CONTROLLER(PHANDLE) CONTROLLER_DATA(3)
interrupt-map = < 0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1
0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1
0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1
0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>;
** ^^^^^^
** This should be a phandle to the interrupt controller
// PCI_DEVICE(3) INT#1()
interrupt-map-mask = <0xf800 0x0 0x0 0x7>;
}I keep thinking of making a pcie-dt.h file with helper macros for all this. :|
FWWI, I like to put a double space between the logically distinct things in the lists.
Jason
PATCH v2 3/3 PCI: ARM: add support for generic PCI host controller
Quoted 18 lines · Jason Gunthorpe
Quoted 1 line · Will Deacon
Not in this case! kvmtool generates the following:
Well, the example is nice so someone can review it..
Quoted 10 lines
pci { #address-cells = <0x3>; #size-cells = <0x2>; #interrupt-cells = <0x1>; compatible = "arm,pci-cam-generic"; reg = <0x0 0x40000000>; ranges = <0x1000000 0x0 0x0 0x0 0x0 0x0 0x10000 0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>; interrupt-map = <0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1 0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1 0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1 0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>; interrupt-map-mask = <0xf800 0x0 0x0 0x7>; };
Quoted 7 lines
Looks like there are a few misses in the above. How about this:
pci {
compatible = "arm,pci-cam-generic"
device_type= "pci";
** ^^^^^^^^^^^^^^^^^
** MANDATORY for the host bridge node
Ok, I'll add that.
Quoted 11 lines
#address-cells = <3>;
#size-cells = <2>;
// BUS_ADDRESS(3) CPU_PHYSICAL(2) SIZE(2)
ranges = <0x1000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00010000>,
^^^^^^^^^^^^^^
** ?? Is this why you had problems with the offset? Should
** be the cpu physical address of the start of the IO window.
<0x2000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>;Quoted 8 lines
#interrupt-cells = <0x1>;
// PCI_DEVICE(3) INT#(1) CONTROLLER(PHANDLE) CONTROLLER_DATA(3)
interrupt-map = < 0x0 0x0 0x0 0x1 0x1 0x0 0x4 0x1
0x800 0x0 0x0 0x1 0x1 0x0 0x5 0x1
0x1000 0x0 0x0 0x1 0x1 0x0 0x6 0x1
0x1800 0x0 0x0 0x1 0x1 0x0 0x7 0x1>;
** ^^^^^^
** This should be a phandle to the interrupt controllerThat is a phandle, libfdt/kvmtool just ends up dealing with the number directly. I'll fix in the example.
Quoted 6 lines
// PCI_DEVICE(3) INT#1() interrupt-map-mask = <0xf800 0x0 0x0 0x7>; }I keep thinking of making a
pcie-dt.hfile with helper macros for all this. :|
If it's not being generated automatically, that could be useful.
Quoted 2 lines
FWWI, I like to put a double space between the logically distinct things in the lists.
Makes sense, I may well do the same.
Thanks,
Will
v2 cover ARM: PCI: implement generic PCI host controller
In reply to Will DeaconQuoted 4 lines · Will Deacon
Like v1, I continue to support only a single controller and therefore a single I/O space. I'm not sure how to represent multiple controllers in the device-tree without inventing some horrible hacks, so any ideas in this area would be much appreciated.
The DT representation is very straightforward, just have more copies of what you already have. Each DT stanza should be represented in Linux a distinct PCI domain.
In Linux you run into two small problems
1) PCI Domain numbers needs to be allocated dynamically
* I think there should be a core thing to allocate a domain object w/ a struct device, and assign a unique domain number. We are already seeing drivers do things like keep track of their own domain numbers via a counter (pcie-designware.c) The host bridge object is similar to this but it isn't focused on a domain.
2) The space in the IO fixed mapping needs to be allocated to PCI host drivers dynamically
* pci_ioremap_io_dynamic that takes a bus address + cpu_physical address and returns a Linux virtual address.
The first caller can get a nice traslation where bus address == Linux virtual address, everyone after can get best efforts.
You will have overlapping physical IO bus addresses - each domain will have a 0 IO BAR - but those will have distinct CPU physical addresses and can then be uniquely mapped into the IO mapping. So at the struct resource level the two domains have disjoint IO addresses, but each domain uses a different IO offset..
Jason
v2 cover ARM: PCI: implement generic PCI host controller
Quoted 4 lines · On Thursday 13 February 2014 11:26:55 Jason Gunthorpe wrote:
The DT representation is very straightforward, just have more copies of what you already have. Each DT stanza should be represented in Linux a distinct PCI domain.
Quoted 8 lines
In Linux you run into two small problems
1) PCI Domain numbers needs to be allocated dynamically
* I think there should be a core thing to allocate a domain object w/ a struct device, and assign a unique domain number. We are already seeing drivers do things like keep track of their own domain numbers via a counter (pcie-designware.c) The host bridge object is similar to this but it isn't focused on a domain.
Right, see also my other comment I just sent in the "Re: [PATCH v2 3/3]
PCI: ARM: add support for generic PCI host controller" thread.
The host driver is the wrong place to pick a domain number, but someone has to do it.
Quoted 6 lines
2) The space in the IO fixed mapping needs to be allocated to PCI host drivers dynamically
*pci_ioremap_io_dynamicthat takes a bus address +cpu_physicaladdress and returns a Linux virtual address.
The first caller can get a nice traslation where bus address == Linux virtual address, everyone after can get best efforts.
I think we can have a helper that everything we need to do with the I/O space:
* parse the ranges property
* pick an appropriate virtual address window
* ioremap the physical window there
* compute the io_offset
* pick a name for the resource
* request the io resource
* register the pci_host_bridge_window
Quoted 5 lines
You will have overlapping physical IO bus addresses - each domain will have a 0 IO BAR - but those will have distinct CPU physical addresses and can then be uniquely mapped into the IO mapping. So at the struct resource level the two domains have disjoint IO addresses, but each domain uses a different IO offset..
This would be the common case, but when we have a generic helper function, it's actually not that are to handle a couple of variations of that, which we may see in the field and can easily be described with the existing binding.
* If we allow multiple host bridges to be in the same PCI domain with a split bus space, we should also allow them to have a split I/O space, e.g. have two 32KB windows, both with io_offset=0. This would imply that the second bridge can only support relocatable I/O BARs.
* Similar to that, you may have multiple 64KB windows with io_offset=0.
* Some system may have hardwire I/O windows at a high bus address larger than IO_SPACE_LIMIT. This would mean a *negative* io_offset. I can't see any reason why anyone would do this, but I also don't see a reason to prevent it if we can easily keep the code generic enough to handle it without adding extra code.
* A more obscure case would be to have multiple I/O windows on a bus. This is allowed by the binding and by the pci_host_bridge_window, and while again I don't see a use case, it also doesn't seem hard to do, we just keep looping for all ranges rather than stop after the first window.
Arnd
v2 cover ARM: PCI: implement generic PCI host controller
Quoted 10 lines · Arnd Bergmann
Quoted 6 lines
2) The space in the IO fixed mapping needs to be allocated to PCI host drivers dynamically
*pci_ioremap_io_dynamicthat takes a bus address +cpu_physicaladdress and returns a Linux virtual address.
The first caller can get a nice traslation where bus address == Linux virtual address, everyone after can get best efforts.I think we can have a helper that everything we need to do with the I/O space:
Quoted 7 lines
* parse the ranges property
* pick an appropriate virtual address window
* ioremap the physical window there
* compute theio_offset
* pick a name for the resource
* request the io resource
* register thepci_host_bridge_window
Sounds good to me
Quoted 10 lines
Quoted 5 lines
You will have overlapping physical IO bus addresses - each domain will have a 0 IO BAR - but those will have distinct CPU physical addresses and can then be uniquely mapped into the IO mapping. So at the struct resource level the two domains have disjoint IO addresses, but each domain uses a different IO offset..
This would be the common case, but when we have a generic helper function, it's actually not that are to handle a couple of variations of that, which we may see in the field and can easily be described with the existing binding.
I agree the DT binding ranges has enough flexibility to describe all of these cases, but I kind of circle back to the domain discussion and ask 'Why?'.
As far as I can see there are two reasonable ways to handle IO space:
- The IO space is 1:1 mapped to the Physical CPU Address. In most cases this would require 32 bit IO BARS in all devices.
- The IO space in a domain is always 0 -> 64k and thus only ever requires 16 bit BARs
And this is possible too:
- The IO space is 1:1: mapped to Linux Virtual IO port numbers (which are a fiction) and devices sometimes require 32 bit IO BARs. This gives you lspci output that matches dmesg and /proc/ioport.
Things get more complex if you want to support legacy non-BAR IO (eg VGA). Then you *really* want every domain to support 0->64k and you need driver support to setup a window for the legacy IO port. (eg on a multi-port root complex there is non-PCI spec hardware that routes the VGA addresses to the root port bridge that connects to the VGA card) Plus you probably need a memory hole around 1M..
But, I think this is overthinking things. IO space really is deprecated, and 0->64k is a fine default for everything but very specialized cases.
Jason
v2 cover ARM: PCI: implement generic PCI host controller
Quoted 12 lines · On Tuesday 18 February 2014 11:00:05 Jason Gunthorpe wrote:
Quoted 11 lines
Quoted 6 lines
You will have overlapping physical IO bus addresses - each domain will have a 0 IO BAR - but those will have distinct CPU physical addresses and can then be uniquely mapped into the IO mapping. So at the struct resource level the two domains have disjoint IO addresses, but each domain uses a different IO offset..
This would be the common case, but when we have a generic helper function, it's actually not that are to handle a couple of variations of that, which we may see in the field and can easily be described with the existing binding.
Quoted 7 lines
I agree the DT binding ranges has enough flexibility to describe all of these cases, but I kind of circle back to the domain discussion and ask 'Why?'.
As far as I can see there are two reasonable ways to handle IO space:
- The IO space is 1:1 mapped to the Physical CPU Address. In most cases this would require 32 bit IO BARS in all devices.
I wouldn't consider this one reasonable ;-)
In particular, it would break /dev/port access horribly.
Quoted 8 lines
- The IO space in a domain is always 0 -> 64k and thus only ever requires 16 bit BARs
And this is possible too:
- The IO space is 1:1: mapped to Linux Virtual IO port numbers (which are a fiction) and devices sometimes require 32 bit IO BARs. This gives you lspci output that matches dmesg and/proc/ioport.
These two seem fine.
Quoted 6 lines
Things get more complex if you want to support legacy non-BAR IO (eg VGA). Then you *really* want every domain to support 0->64k and you need driver support to setup a window for the legacy IO port. (eg on a multi-port root complex there is non-PCI spec hardware that routes the VGA addresses to the root port bridge that connects to the VGA card) Plus you probably need a memory hole around 1M..
For the I/O space part of this, that would be the same as your second example above. For memory space, you need both the 640k-1M window and the 15M-16M window. We have a couple of ARM platforms that actually have a PCI MMIO window starting at bus address 0 with a mem_offset matching the physical base address to facilitate VGA access, try 'git grep -w vga_base'.
Quoted 3 lines
But, I think this is overthinking things. IO space really is deprecated, and 0->64k is a fine default for everything but very specialized cases.
We may not need the entire complexity to handle all cases, but I think it's a good idea to detect the ones we don't handle and then warn about them. For some things, just doing it right may be easier than detecting the case where we don't.
Arnd