PCIe support for the Armada 370 and Armada XP SoCs
Subsystems (10)
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
Hello,
This series of patches introduces PCIe support for the Marvell Armada 370 and Armada XP.
This PATCHv3 follows:
* PATCHv2, sent on January, 28th 2013
* RFCv1, sent on December, 7th 2012
Thanks to all the people who discussed on the previous version of the patch set. The discussions have been long and complicated, but certainly very useful.
In order to make progress on this, and help reducing the size of the patch set, I would ask if it would be possible to merge patches 1 to 17 from the series for 3.9 (only preparation work), keeping the rest for 3.10. The patches in question are PCI-related, ARM-related, and mvebu/orion-related.
Changes between v2 and v3:
* Use of_irq_map_pci() instead of of_irq_map_raw(), as suggested by Andrew Murray. In order to do this, we moved the interrupt-map and interrupt-map-mask DT properties from the main PCIe controller node to the DT subnodes representing each PCIe interface.
* Remove the usage of the emulated host bridge.
* Move the emulated PCI-to-PCI bridge code into the Marvell PCI driver itself, in order to allow a tighter integration. Suggested by Bjorn Helgaas and Jason Gunthorpe.
* Make the allocation of address decoding windows dynamic: it's when memory accesses or I/O accesses are enabled at the PCI-to-PCI bridge level that we allocate and setup the corresponding address decoding window. Requested by Bjorn Helgaas.
* Fixed the implementation of I/O accesses to use I/O addresses that fall within the normal IO_SPACE_LIMIT. This required using the "remap" functionality of address decoding windows, and therefore some changes in the address decoding window allocator. Follows a long discussion about I/O accesses.
* Set up a correct bus number in the configuration of the PCIe interfaces so that we don't have to fake bus numbers anymore. Requested by Jason Gunthorpe.
* Fix the of_pci_get_devfn() implementation according to Stephen Warren's comment.
* Use CFLAGS_ instead of ccflags to add the mach-mvebu and plat-orion include paths when building the pci-mvebu driver. This ensures that the include paths are only added when building this specific driver. Requested by Stephen Warren.
* Fix the ->resource_align() to only apply on bus 0 (the one on which the emulated PCI-to-PCI bridges sit), and to request an alignment on the size of the window (and not only 64 KB for I/O windows and 1 MB for memory windows).
* Clarified the commit log of "clk: mvebu: create parent-child relation for PCIe clocks on Armada 370"
A quick description of the patches:
* Patches 1 to 3 add PCI-related Device Tree parsing functions. Those patches are common with the Nvidia Tegra PCIe patch set from Thierry Redding. They are included in this series so that it can be tested easily.
* Patch 4 extends the ARM PCI core to store a per-controller private data pointer. This patch is common with the Nvidia Tegra PCIe patch set from Thierry Redding. It is included in this series so that it can be tested easily.
* Patch 5 fixes a problem in lib/devres.c that prevents certain PCI-related functions from being visible on NO_IOPORT platforms. I know this patch isn't acceptable by itself, but the discussion about this has been so huge and went in so many directions that in the end, I don't know what is the correct way of fixing this. If an agreement is found on how to fix this properly, I'm willing to work on it if needed.
* Patch 6 extends the ARM PCI core with an additional hook that a PCI controller driver can register and get called to realign PCI ressource addresses. This is needed for the support of Marvell PCIe interfaces because the address decoding windows for I/O ranges have a granularity of 64 KB, while the PCI standard requires only a 4 KB alignement. See the patch itself for details.
* Patch 7 fixes a mistake in the interrupt controller node of the Armada 370/XP Device Tree, which was invisible until we started using the of_irq_map_raw() function, needed in our PCIe support.
* Patches 8 and 9 fix some issues in the Armada 370/XP clock gating driver, related to PCIe interfaces.
* Patches 10 and 11 are cleanup/refactoring of the common plat-orion address decoding code, in preparation for further changes related to PCIe.
* Patches 12 to 17 introduce a ORION_ADDR_MAP_NO_REMAP define that is used by existing Marvell SoC code to say "I don't need this window to remap anything". Previously a -1 value was used as the remap address to communicate the fact that no remap is needed, but this prevents any remap address higher than 2 GB.
* Patch 18 removes __init from a few address window decoding functions that are now needed after boot.
* Patch 19 introduces in the common plat-orion address decoding code functions to allocate/free an address decoding window. Until now, the address decoding windows were configured statically. With Armada XP having up to 10 PCIe interfaces, we don't want to allocate useless address decoding windows statically, so we move to a more dynamic model in which address decoding windows are configured only for the PCIe interfaces that are actually in use.
* Patch 20 removes __init from a few PCIe functions that are now needed after boot.
* Patch 21 improves the Armada 370/XP specific address decoding code to provide functions that add and remove an address decoding window for a given PCIe interface. It relies on the common functions added in patch 19.
* Patch 22 makes the common plat-orion PCIe code available on PLAT_ORION platforms such as ARCH_MVEBU.
* Patch 23 creates the drivers/pci/host directory and makes the related minimal changes to Kconfig/Makefile. This patch will trivially conflict with the NVidia Tegra PCIe support posted by Thierry Redding, which also creates the drivers/pci/host directory.
* Patch 24 contains the Armada 370/XP PCIe driver itself, that implements the necessary operations required by the ARM PCI core, and configures the address decoding windows as needed. This driver relies on a Device Tree description of the PCIe interfaces.
* Patch 25 marks the ARCH_MVEBU platform has having PCI available, which allows the compilation of the PCIe support.
* Patches 26 and 27 add the SoC-level Device Tree informations related to PCIe for Armada 370 and Armada XP.
* Patch 28 to 31 add the board-level Device Tree informations related to PCIe for the Armada XP DB, Armada 370 DB, PlatHome OpenBlocks AX3-4 and GlobalScale Mirabox boards.
* Patch 32 updates mvebu_defconfig with PCI and USB support.
This patch set applies on top of v3.8-rc7, and has been pushed at:
git://github.com/MISL-EBU-System-SW/mainline-public.git marvell-pcie-v3
Thanks,
Thomas
---
Output of lspci -vvv:
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00010000-00010fff
Memory behind bridge: c1000000-c10fffff
Prefetchable memory behind bridge: c1100000-c11fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: 00020000-00020fff
Memory behind bridge: c1200000-c12fffff
Prefetchable memory behind bridge: c1300000-c13fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
03:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
Subsystem: Marvell Technology Group Ltd. Device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 105
Region 0: Memory at c1000000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at 10000 [size=256]
[virtual] Expansion ROM@c1100000 [disabled] [size=512K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000012345678 Data: 0000
Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <256ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 <256ns, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: sata_mv
05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 106
Region 0: Memory at c1200000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at c1220000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 20000 [disabled] [size=32]
[virtual] Expansion ROM@c1300000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-c1-c4-fe
Kernel driver in use: e1000e
Boot messages from the PCI subsystem:
mvebu-pcie pcie-controller.1: PCIe0.0: link down
mvebu-pcie pcie-controller.1: PCIe0.1: link down
mvebu-pcie pcie-controller.1: PCIe0.2: link up
mvebu-pcie pcie-controller.1: PCIe0.3: link down
mvebu-pcie pcie-controller.1: PCIe2.0: link up
mvebu-pcie pcie-controller.1: PCIe3.0: link down
mvebu-pcie pcie-controller.1: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0x0000-0xfffff]pci_bus 0000:00: root bus resource [mem 0xc1000000-0xc8ffffff] pci_bus 0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers enabled
PCI: bus2: Fast back to back transfers enabled
PCI: bus3: Fast back to back transfers disabled
PCI: bus4: Fast back to back transfers enabled
PCI: bus5: Fast back to back transfers disabled
PCI: bus6: Fast back to back transfers enabled
pci 0000:00:03.0: BAR 8: assigned [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: BAR 9: assigned [mem 0xc1100000-0xc11fffff pref] pci 0000:00:05.0: BAR 8: assigned [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: BAR 9: assigned [mem 0xc1300000-0xc13fffff pref] pci 0000:00:03.0: BAR 7: assigned [io 0x10000-0x10fff]
pci 0000:00:05.0: BAR 7: assigned [io 0x20000-0x20fff]
pci 0000:00:01.0: PCI bridge to [bus 01]
pci 0000:00:02.0: PCI bridge to [bus 02]
pci 0000:03:00.0: BAR 0: assigned [mem 0xc1000000-0xc10fffff 64bit] pci 0000:03:00.0: BAR 6: assigned [mem 0xc1100000-0xc117ffff pref] pci 0000:03:00.0: BAR 2: assigned [io 0x10000-0x100ff]
pci 0000:00:03.0: PCI bridge to [bus 03]
pci 0000:00:03.0: bridge window [io 0x10000-0x10fff] pci 0000:00:03.0: bridge window [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: bridge window [mem 0xc1100000-0xc11fffff pref]
pci 0000:00:04.0: PCI bridge to [bus 04]
pci 0000:05:00.0: BAR 0: assigned [mem 0xc1200000-0xc121ffff] pci 0000:05:00.0: BAR 1: assigned [mem 0xc1220000-0xc123ffff] pci 0000:05:00.0: BAR 6: assigned [mem 0xc1300000-0xc131ffff pref] pci 0000:05:00.0: BAR 2: assigned [io 0x20000-0x2001f] pci 0000:00:05.0: PCI bridge to [bus 05]
pci 0000:00:05.0: bridge window [io 0x20000-0x20fff] pci 0000:00:05.0: bridge window [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: bridge window [mem 0xc1300000-0xc13fffff pref]
pci 0000:00:06.0: PCI bridge to [bus 06]
PCI: enabling device 0000:00:01.0 (0140 -> 0143)
PCI: enabling device 0000:00:02.0 (0140 -> 0143)
PCI: enabling device 0000:00:03.0 (0140 -> 0143)
PCI: enabling device 0000:00:04.0 (0140 -> 0143)
PCI: enabling device 0000:00:05.0 (0140 -> 0143)
PCI: enabling device 0000:00:06.0 (0140 -> 0143)
PATCH 1/32 of/pci: Provide support for parsing PCI DT ranges property
From: Andrew Murray <[email protected]>
DT bindings for PCI host bridges often use the ranges property to describe memory and IO ranges - this binding tends to be the same across architectures yet several parsing implementations exist, e.g. arch/mips/pci/pci.c, arch/powerpc/kernel/pci-common.c, arch/sparc/kernel/pci.c and arch/microblaze/pci/pci-common.c (clone of PPC). Some of these duplicate functionality provided by drivers/of/address.c.
This patch provides a common iterator-based parser for the ranges property, it is hoped this will reduce DT representation differences between architectures and that architectures will migrate in part to this new parser.
It is also hoped (and the motativation for the patch) that this patch will reduce duplication of code when writing host bridge drivers that are supported by multiple architectures.
This patch provides struct resources from a device tree node, e.g.:
u32 *last = NULL;
struct resource res;
while ((last = of_pci_process_ranges(np, res, last))) {
//do something with res
}Platforms with quirks can then do what they like with the resource or migrate common quirk handling to the parser. In an ideal world drivers can just request the obtained resources and pass them on (e.g. pci_add_resource_offset).
--- drivers/of/address.c | 63 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/of_address.h | 9 +++++++ 2 files changed, 72 insertions(+)
drivers/of/address.c +63 -0
@@ -13,6 +13,7 @@#define OF_CHECK_COUNTS (na, ns) (OF_CHECK_ADDR_COUNT(na) && (ns) > 0)static struct of_bus *of_match_bus (struct device_node *np); +static struct of_bus *of_find_bus (const char *name);static int __of_address_to_resource (struct device_node *dev,const __be32 *addrp,u64 size,unsigned int flags,const char *name,struct resource *r);
@@ -227,6 +228,57 @@ int of_pci_address_to_resource(struct device_node *dev, int bar,return __of_address_to_resource (dev, addrp, size, flags,NULL , r); }EXPORT_SYMBOL_GPL (of_pci_address_to_resource); + +const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res,const __be32 *from) +{ +const __be32 *start, *end; +int na, ns, np, pna; +int rlen; +struct of_bus *bus; + +WARN_ON (!res); + + bus =of_find_bus ("pci" ); + bus->count_cells (node, &na, &ns); +if (!OF_CHECK_COUNTS (na, ns)) { +pr_err ("Bad cell count for %s\n" , node->full_name); +return NULL ; + } + + pna =of_n_addr_cells (node); + np = pna + na + ns; + + start =of_get_property (node,"ranges" , &rlen); +if (start ==NULL ) +return NULL ; + + end = start + rlen /sizeof (__be32); + +if (!from) + from = start; + +while (from + np <= end) { +u64 cpu_addr, size; + + cpu_addr =of_translate_address (node, from + na); + size =of_read_number (from + na + pna, ns); + res->flags = bus->get_flags (from); + from += np; + +if (cpu_addr ==OF_BAD_ADDR || size ==0 ) +continue ; + + res->name = node->full_name; + res->start = cpu_addr; + res->end = res->start + size -1 ; + res->parent = res->child = res->sibling =NULL ; +return from; + } + +return NULL ; +} +EXPORT_SYMBOL_GPL (of_pci_process_ranges);#endif /* CONFIG_PCI */ /*
@@ -337,6 +389,17 @@ static struct of_bus *of_match_bus(struct device_node *np)return NULL ; } +static struct of_bus *of_find_bus (const char *name) +{ +unsigned int i; + +for (i =0 ; i <ARRAY_SIZE (of_busses); i++) +if (strcmp (name, of_busses[i].name) ==0 ) +return &of_busses[i]; + +return NULL ; +} +static int of_translate_one (struct device_node *parent,struct of_bus *bus,struct of_bus *pbus,__be32 *addr,int na,int ns,int pna,const char *rprop)
include/linux/of_address.h +9 -0
@@ -27,6 +27,8 @@ static inline unsigned long pci_address_to_pio(phys_addr_t addr) { return -1; }#define pci_address_to_pio pci_address_to_pio#endif +const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res,const __be32 *from);#else /* CONFIG_OF_ADDRESS */ #ifndef of_address_to_resource static inline int of_address_to_resource(struct device_node *dev, int index,
@@ -53,6 +55,13 @@ static inline const __be32 *of_get_address(struct device_node *dev, int index, {return NULL ; } + +static inline const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res, +const __be32 *from) +{ +return NULL ; +}#endif /* CONFIG_OF_ADDRESS */
Signature · 1 line
1.7.9.5
PATCH 2/32 of/pci: Add of_pci_get_devfn() function
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
This function can be used to parse the device and function number from a standard 5-cell PCI resource. PCI_SLOT() and PCI_FUNC() can be used on the returned value obtain the device and function numbers respectively.
---
v2 -> v2:
As suggested by Stephen Warren, rename the 'devfn' argument to 'data', and use a local variable named 'devfn' instead of 'err', which was a quite confusing name indeed.
--- drivers/of/of_pci.c | 34 +++++++++++++++++++++++++++++----- include/linux/of_pci.h | 1 + 2 files changed, 30 insertions(+), 5 deletions(-)
drivers/of/of_pci.c +29 -5
@@ -5,14 +5,15 @@#include <asm/prom.h> static inline int __of_pci_pci_compare (struct device_node *node, -unsigned int devfn) +unsigned int data) { -unsigned int size; -const __be32 *reg =of_get_property (node,"reg" , &size); +int devfn; -if (!reg || size <5 *sizeof (__be32)) + devfn =of_pci_get_devfn (node); +if (devfn <0 )return 0 ; -return ((be32_to_cpup (®[0 ]) >>8 ) &0xff ) == devfn; + +return data == devfn; } struct device_node *of_pci_find_child_device(struct device_node *parent,
@@ -40,3 +41,26 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,return NULL ; }EXPORT_SYMBOL_GPL (of_pci_find_child_device); + +/** +* of_pci_get_devfn() - Get device and function numbers for a device node +* @np: device node +* +* Parses a standard 5-cell PCI resource and returns an 8-bit value that can +* be passed to the PCI_SLOT() and PCI_FUNC() macros to extract the device +* and function numbers respectively. On error a negative error code is +* returned. +*/ +int of_pci_get_devfn (struct device_node *np) +{ +unsigned int size; +const __be32 *reg; + + reg =of_get_property (np,"reg" , &size); + +if (!reg || size <5 *sizeof (__be32)) +return -EINVAL ; + +return (be32_to_cpup (reg) >>8 ) &0xff ; +} +EXPORT_SYMBOL_GPL (of_pci_get_devfn);
include/linux/of_pci.h +1 -0
@@ -10,5 +10,6 @@ int of_irq_map_pci(const struct pci_dev *pdev, struct of_irq *out_irq);struct device_node ;struct device_node *of_pci_find_child_device (struct device_node *parent,unsigned int devfn); +int of_pci_get_devfn (struct device_node *np);#endif
Signature · 1 line
1.7.9.5
PATCH 3/32 of/pci: Add of_pci_parse_bus_range() function
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
This function can be used to parse a bus-range property as specified by device nodes representing PCI bridges.
--- drivers/of/of_pci.c | 25 +++++++++++++++++++++++++ include/linux/of_pci.h | 1 + 2 files changed, 26 insertions(+)
drivers/of/of_pci.c +25 -0
@@ -64,3 +64,28 @@ int of_pci_get_devfn(struct device_node *np)return (be32_to_cpup (reg) >>8 ) &0xff ; }EXPORT_SYMBOL_GPL (of_pci_get_devfn); + +/** +* of_pci_parse_bus_range() - parse the bus-range property of a PCI device +* @node: device node +* @res: address to a struct resource to return the bus-range +* +* Returns 0 on success or a negative error-code on failure. +*/ +int of_pci_parse_bus_range (struct device_node *node,struct resource *res) +{ +const __be32 *values; +int len; + + values =of_get_property (node,"bus-range" , &len); +if (!values || len <sizeof (*values) *2 ) +return -EINVAL ; + + res->name = node->name; + res->start =be32_to_cpup (values++); + res->end =be32_to_cpup (values); + res->flags =IORESOURCE_BUS ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (of_pci_parse_bus_range);
include/linux/of_pci.h +1 -0
@@ -11,5 +11,6 @@ struct device_node;struct device_node *of_pci_find_child_device (struct device_node *parent,unsigned int devfn);int of_pci_get_devfn (struct device_node *np); +int of_pci_parse_bus_range (struct device_node *node,struct resource *res);#endif
Signature · 1 line
1.7.9.5
PATCH 4/32 ARM: pci: Allow passing per-controller private data
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
In order to allow drivers to specify private data for each controller, this commit adds a private_data field to the struct hw_pci. This field is an array of nr_controllers pointers that will be used to initialize the private_data field of the corresponding controller's pci_sys_data structure.
--- arch/arm/include/asm/mach/pci.h | 1 + arch/arm/kernel/bios32.c | 3 +++ 2 files changed, 4 insertions(+)
arch/arm/include/asm/mach/pci.h +1 -0
@@ -23,6 +23,7 @@ struct hw_pci {#endif struct pci_ops *ops;int nr_controllers; +void **private_data;int (*setup)(int nr,struct pci_sys_data *);struct pci_bus *(*scan)(int nr,struct pci_sys_data *);void (*preinit)(void );
arch/arm/kernel/bios32.c +3 -0
@@ -464,6 +464,9 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->map_irq = hw->map_irq;INIT_LIST_HEAD (&sys->resources); +if (hw->private_data) + sys->private_data = hw->private_data[nr]; + ret = hw->setup (nr, sys);if (ret >0 ) {
Signature · 1 line
1.7.9.5
PATCH 5/32 lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
In reply to Thomas PetazzoniThe pcim_*() functions are used by the libata-sff subsystem, and this subsystem is used for many SATA drivers on ARM platforms that do not necessarily have I/O ports.
--- lib/devres.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/devres.c +1 -1
@@ -195,6 +195,7 @@ void devm_ioport_unmap(struct device *dev, void __iomem *addr) devm_ioport_map_match, (void *)addr)); }EXPORT_SYMBOL (devm_ioport_unmap); +#endif /* CONFIG_HAS_IOPORT */ #ifdef CONFIG_PCI /*
@@ -400,4 +401,3 @@ void pcim_iounmap_regions(struct pci_dev *pdev, int mask) }EXPORT_SYMBOL (pcim_iounmap_regions);#endif /* CONFIG_PCI */ -#endif /* CONFIG_HAS_IOPORT */
Signature · 1 line
1.7.9.5
Re: [PATCH 05/32] lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
Quoted 9 lines · Thomas Petazzoni
The pcim_*() functions are used by the libata-sff subsystem, and this subsystem is used for many SATA drivers on ARM platforms that do not necessarily have I/O ports.
Signed-off-by Thomas Petazzoni <[email protected]>Cc Paul Gortmaker <[email protected]>Cc Jesse Barnes <[email protected]>Cc Yinghai Lu <[email protected]>Cc linux-kernel at vger.kernel.org
Sorry, but this patch is still incorrect. Any driver that requires a linear mapping of I/O ports to __iomem pointers must depend CONFIG_HAS_IOPORT with the current definition of that symbol (as mentioned before, we should really rename that to CONFIG_HAS_IOPORT_MAP). Having these functions not defined is a compile time check that is necessary to ensure that all drivers have the correct annotation.
If a platform has no support for I/O ports at all, it should probably not set CONFIG_NO_IOPORT at this point.
Arnd
Re: [PATCH 05/32] lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
Dear Arnd Bergmann,
Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · Thomas Petazzoni
The pcim_*() functions are used by the libata-sff subsystem, and this subsystem is used for many SATA drivers on ARM platforms that do not necessarily have I/O ports.
Quoted 7 lines
Quoted 5 lines
Signed-off-by: Thomas Petazzoni <[email protected]> Cc: Paul Gortmaker <[email protected]> Cc: Jesse Barnes <[email protected]> Cc: Yinghai Lu <[email protected]>
Cc: linux-kernel at vger.kernel.orgSorry, but this patch is still incorrect.
I know, but the discussion was so huge on the first posting that it was basically impossible to draw a conclusion out of it.
Quoted 7 lines
Any driver that requires a
linear mapping of I/O ports to__iomempointers must dependCONFIG_HAS_IOPORTwith the current definition of that symbol (as mentioned before, we should really rename that toCONFIG_HAS_IOPORT_MAP). Having these functions not defined is a compile time check that is necessary to ensure that all drivers have the correct annotation.
I have the feeling that the problem is more complex than that. My understanding is that the pcim_iomap_regions() function used by drivers/ata/libata-sff.c can perfectly be used to map memory BARs, and not necessarily I/O BARs. Therefore, this driver can perfectly be used in an architecture where CONFIG_NO_IOPORT is selected.
The thing is that pcim_iomap_regions() transparently allows to remap an I/O BAR is such a BAR is passed as argument, or a memory BAR if such a BAR is passed as argument.
Therefore, I continue to believe that the pcim_*() functions are useful even if the platform doesn't have CONFIG_HAS_IOPORT.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Re: [PATCH 05/32] lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
Quoted 8 lines · Thomas Petazzoni
Quoted 7 lines
Any driver that requires a
linear mapping of I/O ports to__iomempointers must dependCONFIG_HAS_IOPORTwith the current definition of that symbol (as mentioned before, we should really rename that toCONFIG_HAS_IOPORT_MAP). Having these functions not defined is a compile time check that is necessary to ensure that all drivers have the correct annotation.
Quoted 5 lines
I have the feeling that the problem is more complex than that. My understanding is that the
pcim_iomap_regions()function used bydrivers/ata/libata-sff.ccan perfectly be used to map memory BARs, and not necessarily I/O BARs. Therefore, this driver can perfectly be used in an architecture whereCONFIG_NO_IOPORTis selected.
That is correct.
Quoted 6 lines
The thing is that
pcim_iomap_regions()transparently allows to remap an I/O BAR is such a BAR is passed as argument, or a memory BAR if such a BAR is passed as argument.Therefore, I continue to believe that the pcim_*() functions are useful even if the platform doesn't have
CONFIG_HAS_IOPORT.
Yes, the pcim_ functions are useful in principle, but it falls back to the __pci_ioport_map() for IORESOURCE_IO, and that needs to return an error if CONFIG_HAS_IOPORT is not set.
I think it would be correct if you add this hunk:
lib/pci_iomap.c +1 -1
@@ -33,7 +33,7 @@ void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)return NULL ;if (maxlen && len > maxlen) len = maxlen; -if (flags &IORESOURCE_IO ) +if (IS_ENABLED (CONFIG_HAS_IOPORT ) && (flags &IORESOURCE_IO ))return __pci_ioport_map (dev, start, len);if (flags &IORESOURCE_MEM ) {if (flags &IORESOURCE_CACHEABLE )
in order to prevent a link error when CONFIG_HAS_IOPORT is unset.
Arnd
Re: [PATCH 05/32] lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
Dear Arnd Bergmann,
Quoted 34 lines · Arnd Bergmann
Quoted 5 lines
I have the feeling that the problem is more complex than that. My understanding is that the
pcim_iomap_regions()function used bydrivers/ata/libata-sff.ccan perfectly be used to map memory BARs, and not necessarily I/O BARs. Therefore, this driver can perfectly be used in an architecture whereCONFIG_NO_IOPORTis selected.That is correct.
Quoted 6 lines
The thing is that
pcim_iomap_regions()transparently allows to remap an I/O BAR is such a BAR is passed as argument, or a memory BAR if such a BAR is passed as argument.Therefore, I continue to believe that the pcim_*() functions are useful even if the platform doesn't have
CONFIG_HAS_IOPORT.Yes, the
pcim_functions are useful in principle, but it falls back to the__pci_ioport_map()forIORESOURCE_IO, and that needs to return an error ifCONFIG_HAS_IOPORTis not set.
I think it would be correct if you add this hunk:lib/pci_iomap.c +1
-1@@ -33,7 +33,7 @@ void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)return NULL ;if (maxlen && len > maxlen) len = maxlen; -if (flags &IORESOURCE_IO ) +if (IS_ENABLED (CONFIG_HAS_IOPORT ) && (flags &IORESOURCE_IO ))return __pci_ioport_map (dev, start, len);if (flags &IORESOURCE_MEM ) {if (flags &IORESOURCE_CACHEABLE )
Quoted 1 line
in order to prevent a link error when
CONFIG_HAS_IOPORTis unset.
FWIW, a patch that is doing what I was initially proposing has been merged for 3.9, and it doesn't contain the IS_ENABLED(CONFIG_HAS_IOPORT) test you were proposing (and which I think was correct). See:
commit 9ed8a30f3471347c1b763bd062fa78ae80f18eae
Author: Jingoo Han <[email protected]> Date: Wed Feb 27 17:02:42 2013 -0800
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Re: [PATCH 05/32] lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
Quoted 9 lines · Thomas Petazzoni
FWIW, a patch that is doing what I was initially proposing has been merged for 3.9, and it doesn't contain the
IS_ENABLED(CONFIG_HAS_IOPORT)test you were proposing (and which I think was correct). See:commit
9ed8a30f3471347c1b763bd062fa78ae80f18eaeAuthor: Jingoo Han <[email protected]> Date: Wed Feb 27 17:02:42 2013 -0800
Sigh.
I'll take it as an additional incentive to finally clean up the logic behind CONFIG_HAS_IOPORT by introducing a CONFIG_HAS_IOPORT_MAP symbol to replace it.
Thanks for the heads up.
Arnd
PATCH 6/32 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniThe PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device), and those have special requirements compared to the standard PCI-to-PCI bridge specifications.
PCIe memory windows must have a power of two size (which matches the PCI spec), be at least of 1 MB (which matches the PCI spec), and be aligned on their size (which does not match the PCI spec, that requires only a 1 MB alignment).
PCIe I/O windows must have a power of two size (matches the PCI spec), be at least of 64 KB (doesn't match the PCI spec, which requires only a size of 4 KB), and be aligned on their size (doesn't match the PCI spec).
The PCI core already calls pcibios_align_resource() in the ARM PCI core, specifically for such purposes. So this patch extends the ARM PCI core so that it calls a ->align_resource() hook registered by the PCI driver, exactly like the existing ->map_irq() and ->swizzle() hooks.
A particular PCI driver can register a align_resource() hook, and do its own specific alignement, depending on the specific constraints of the underlying hardware.
Following the earlier reviews of this patch, a few comments:
* Using numerical values is not possible since the alignment value is not absolute, but depends on the size of the memory window or I/O window. We therefore really need a function hook.
* Using window_alignment() doesn't work, because it doesn't allow to adjust the starting address of the memory or I/O window, and we don't have access to the size. The comment was that ->resource_align() would apply to all resources on all busses, but in the new implementation of the Marvell-specific ->resource_align() hook we are careful to only apply the alignment constraints on the specific bus that hosts the emulated PCI-to-PCI bridges needed to operate the Marvell PCIe interfaces.
--- arch/arm/include/asm/mach/pci.h | 11 +++++++++++ arch/arm/kernel/bios32.c | 6 ++++++ 2 files changed, 17 insertions(+)
arch/arm/include/asm/mach/pci.h +11 -0
@@ -30,6 +30,11 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align); };/*
@@ -51,6 +56,12 @@ struct pci_sys_data {u8 (*swizzle)(struct pci_dev *,u8 *);/* IRQ mapping */ int (*map_irq)(const struct pci_dev *,u8 ,u8 ); +/* Resource alignement requirements */ +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align);void *private_data;/* platform controller private data */ };
arch/arm/kernel/bios32.c +6 -0
@@ -462,6 +462,7 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->busnr = busnr; sys->swizzle = hw->swizzle; sys->map_irq = hw->map_irq; + sys->align_resource = hw->align_resource;INIT_LIST_HEAD (&sys->resources);if (hw->private_data)
@@ -574,6 +575,8 @@ char * __init pcibios_setup(char *str)resource_size_t pcibios_align_resource (void *data,const struct resource *res,resource_size_t size,resource_size_t align) { +struct pci_dev *dev = data; +struct pci_sys_data *sys = dev->sysdata;resource_size_t start = res->start;if (res->flags &IORESOURCE_IO && start &0x300 )
@@ -581,6 +584,9 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res, start = (start + align -1 ) & ~(align -1 ); +if (sys->align_resource) +return sys->align_resource (dev, res, start, size, align); +return start; }
Signature · 1 line
1.7.9.5
PATCH 6/32 arm: pci: add a align_resource hook
Quoted 7 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device), and those have special requirements compared to the standard PCI-to-PCI bridge specifications.
I'm not convince that we should add this complexity yet, until everyone agrees on the basic approach taken.
Arnd
PATCH 6/32 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 11 lines · Arnd Bergmann
Quoted 7 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device), and those have special requirements compared to the standard PCI-to-PCI bridge specifications.
I'm not convince that we should add this complexity yet, until everyone agrees on the basic approach taken.
Regardless of whether we choose to have the emulated PCI-to-PCI bridges or not, we still need this align_resource() hook.
In the solution you propose, where each PCIe interface is represented as a separate PCIe domain, we still need the kernel to dynamically assign ranges of address to each memory BAR and I/O BAR of each PCIe device. And those range of address must comply with the address decoding windows requirements, otherwise, we don't be able to create those address decoding windows, and the devices will be unaccessible.
Of course, if you have an alternate solution to do a _dynamic_ assignment of address ranges to the different PCIe devices, I'm entirely open to suggestions.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 6/32 arm: pci: add a align_resource hook
Quoted 18 lines · Thomas Petazzoni
Dear Arnd Bergmann,
Quoted 11 lines · Arnd Bergmann
Quoted 7 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device), and those have special requirements compared to the standard PCI-to-PCI bridge specifications.
I'm not convince that we should add this complexity yet, until everyone agrees on the basic approach taken.
Regardless of whether we choose to have the emulated PCI-to-PCI bridges or not, we still need this
align_resource()hook.
Quoted 6 lines
In the solution you propose, where each PCIe interface is represented as a separate PCIe domain, we still need the kernel to dynamically assign ranges of address to each memory BAR and I/O BAR of each PCIe device. And those range of address must comply with the address decoding windows requirements, otherwise, we don't be able to create those address decoding windows, and the devices will be unaccessible.
I think what Arnd is suggesting is that you treat each of your busses as a separate root bus.
This then means that you register each bus separately via pci_common_init() (this can take 0..N root buses and scan them.)
We can then size each bus (which means, totalling up the amount of space each bus requires).
At that point, we know how much memory and IO space each bus requires, and platform code can then setup the appropriate bridge windows. The alignment of the bases is something that platform code can deal with in its allocation mechanism, and the resulting allocated resources can then be assigned to each PCI bus.
Once that's done, we can then call pci_bus_assign_resources() to setup the resources for each bus, which will use our allocated resources.
There's a downside to this though: the pci core doesn't size root buses, because:
(a) it assumes that they're fixed size.
(b) root buses don't have a P2P 'bridge' to be sized.
I'm wondering if we could come up with a way to refactor __pci_bus_size_bridges() so that we can get this information out of it. That's more a question for PCI people though.
PATCH 7/32 arm: mvebu: fix address-cells in mpic DT node
In reply to Thomas PetazzoniThere is no need to have a #address-cells property in the MPIC Device Tree node, and more than that, having it confuses the of_irq_map_raw() logic, which will be used by the Marvell PCIe driver.
--- arch/arm/boot/dts/armada-370-xp.dtsi | 1 - 1 file changed, 1 deletion(-)
arch/arm/boot/dts/armada-370-xp.dtsi +0 -1
@@ -31,7 +31,6 @@ mpic: interrupt-controller at d0020000 { compatible = "marvell,mpic"; #interrupt-cells = <1>; - #address-cells = <1>; #size-cells = <1>; interrupt-controller; };
Signature · 1 line
1.7.9.5
PATCH 8/32 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
In reply to Thomas PetazzoniThe Armada 370 has two gatable clocks for each PCIe interface: pexY and pexY_en. In order for a PCIe interface to be functional, we need both of them to be enabled. Since there are no conditions in which having one clock enabled and not the other would be useful, we simply make those two clocks have a child-parent relationship.
So, the pex0 clock becomes a child of pex0_en (for the PCIe interface 0), and pex1 becomes a child of pex1_en (for the PCIe interface 1). This way, the DT node for the PCIe interface 0 only needs to grab a reference to pex0, and the DT node for the PCIe interface 1 only needs to grab a reference to pex1.
--- drivers/clk/mvebu/clk-gating-ctrl.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/clk/mvebu/clk-gating-ctrl.c +2 -2
@@ -119,8 +119,8 @@ static const struct mvebu_soc_descr __initconst armada_370_gating_descr[] = { { "pex1_en", NULL, 2 }, { "ge1", NULL, 3 }, { "ge0", NULL, 4 }, - { "pex0", NULL, 5 }, - { "pex1", NULL, 9 }, + { "pex0", "pex0_en", 5 }, + { "pex1", "pex1_en", 9 }, { "sata0", NULL, 15 }, { "sdio", NULL, 17 }, { "tdm", NULL, 25 },
Signature · 1 line
1.7.9.5
PATCH 9/32 clk: mvebu: add more PCIe clocks for Armada XP
In reply to Thomas PetazzoniThe current revision of the datasheet only mentions the gatable clocks for the PCIe 0.0, 0.1, 0.2 and 0.3 interfaces, and forgot to mention the ones for the PCIe 1.0, 1.1, 1.2, 1.3, 2.0 and 3.0 interfaces. After confirmation with Marvell engineers, this patch adds the missing gatable clocks for those PCIe interfaces.
It also changes the name of the previously existing PCIe gatable clocks, in order to match the naming using the datasheets.
--- drivers/clk/mvebu/clk-gating-ctrl.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/clk/mvebu/clk-gating-ctrl.c +10 -4
@@ -137,10 +137,14 @@ static const struct mvebu_soc_descr __initconst armada_xp_gating_descr[] = { { "ge2", NULL, 2 }, { "ge1", NULL, 3 }, { "ge0", NULL, 4 }, - { "pex0", NULL, 5 }, - { "pex1", NULL, 6 }, - { "pex2", NULL, 7 }, - { "pex3", NULL, 8 }, + { "pex00", NULL, 5 }, + { "pex01", NULL, 6 }, + { "pex02", NULL, 7 }, + { "pex03", NULL, 8 }, + { "pex10", NULL, 9 }, + { "pex11", NULL, 10 }, + { "pex12", NULL, 11 }, + { "pex13", NULL, 12 }, { "bp", NULL, 13 }, { "sata0lnk", NULL, 14 }, { "sata0", "sata0lnk", 15 },
@@ -152,6 +156,8 @@ static const struct mvebu_soc_descr __initconst armada_xp_gating_descr[] = { { "xor0", NULL, 22 }, { "crypto", NULL, 23 }, { "tdm", NULL, 25 }, + { "pex20", NULL, 26 }, + { "pex30", NULL, 27 }, { "xor1", NULL, 28 }, { "sata1lnk", NULL, 29 }, { "sata1", "sata1lnk", 30 },
Signature · 1 line
1.7.9.5
PATCH 10/32 arm: plat-orion: introduce WIN_CTRL_ENABLE in address mapping code
In reply to Thomas PetazzoniInstead of hardcoding "1" as being the bit value to enable an address decoding window, introduce and use a WIN_CTRL_ENABLE definition.
--- arch/arm/plat-orion/addr-map.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm/plat-orion/addr-map.c +3 -1
@@ -38,6 +38,7 @@ EXPORT_SYMBOL_GPL(mv_mbus_dram_info);* CPU Address Decode Windows registers */ #define WIN_CTRL_OFF 0x0000 +#define WIN_CTRL_ENABLE BIT(0)#define WIN_BASE_OFF 0x0004#define WIN_REMAP_LO_OFF 0x0008#define WIN_REMAP_HI_OFF 0x000c
@@ -79,7 +80,8 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, } base_high = base &0xffff0000 ; - ctrl = ((size -1 ) &0xffff0000 ) | (attr <<8 ) | (target <<4 ) |1 ; + ctrl = ((size -1 ) &0xffff0000 ) | (attr <<8 ) | (target <<4 ) | +WIN_CTRL_ENABLE ;writel (base_high, addr +WIN_BASE_OFF );writel (ctrl, addr +WIN_CTRL_OFF );
Signature · 1 line
1.7.9.5
PATCH 11/32 arm: plat-orion: refactor the orion_disable_wins() function
In reply to Thomas PetazzoniIn the address decoding code, the orion_disable_wins() function is used at boot time to disable all address decoding windows, before configuring only the ones that are needed. This allows to make sure that no configuration is left from the bootloader.
As a preparation for the introduction of address decoding window allocation/deallocation function, we refactor this function into an orion_disable_cpu_win() which disables a single window.
The orion_config_wins() function is changed to call orion_disable_cpu_win() in a loop, to preserve an identical behavior.
--- arch/arm/plat-orion/addr-map.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
arch/arm/plat-orion/addr-map.c +17 -18
@@ -95,6 +95,19 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, } } +static void __init orion_disable_cpu_win (const struct orion_addr_map_cfg *cfg, +const int win) +{ +void __iomem *addr = cfg->win_cfg_base (cfg, win); + +writel (0 , addr +WIN_BASE_OFF ); +writel (0 , addr +WIN_CTRL_OFF ); +if (cfg->cpu_win_can_remap (cfg, win)) { +writel (0 , addr +WIN_REMAP_LO_OFF ); +writel (0 , addr +WIN_REMAP_HI_OFF ); + } +} +/* * Configure a number of windows. */
@@ -108,36 +121,22 @@ static void __init orion_setup_cpu_wins(const struct orion_addr_map_cfg * cfg, } } -static void __init orion_disable_wins (const struct orion_addr_map_cfg * cfg) -{ -void __iomem *addr; -int i; - -for (i =0 ; i < cfg->num_wins; i++) { - addr = cfg->win_cfg_base (cfg, i); - -writel (0 , addr +WIN_BASE_OFF ); -writel (0 , addr +WIN_CTRL_OFF ); -if (cfg->cpu_win_can_remap (cfg, i)) { -writel (0 , addr +WIN_REMAP_LO_OFF ); -writel (0 , addr +WIN_REMAP_HI_OFF ); - } - } -} -/* * Disable, clear and configure windows. */ void __init orion_config_wins (struct orion_addr_map_cfg * cfg,const struct orion_addr_map_info *info) { +int win; +if (!cfg->cpu_win_can_remap) cfg->cpu_win_can_remap = orion_cpu_win_can_remap;if (!cfg->win_cfg_base) cfg->win_cfg_base = orion_win_cfg_base; -orion_disable_wins (cfg); +for (win =0 ; win < cfg->num_wins; win++) +orion_disable_cpu_win (cfg, win);if (info)orion_setup_cpu_wins (cfg, info);
Signature · 1 line
1.7.9.5
PATCH 12/32 plat-orion: introduce ORION_ADDR_MAP_NO_REMAP
In reply to Thomas PetazzoniCurrently, the addr_map_info structure uses an "int remap" to allow SoC-specific code to specify whether a given address decoding window should be remapped, and if so, the remapping address. To do so, it tests if the int is less than zero (no remap) or greater than zero. Unfortunately, this prevents any remapping address that is higher than 2 GB.
As a first step to fix this, we introduce ORION_ADDR_MAP_NO_REMAP define, that will be the "remap" value that SoC-specific code should pass to not have a remapping.
--- arch/arm/plat-orion/include/plat/addr-map.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm/plat-orion/include/plat/addr-map.h +2 -0
@@ -41,6 +41,8 @@ struct orion_addr_map_info {const int remap; }; +#define ORION_ADDR_MAP_NO_REMAP (-1) +void __init orion_config_wins (struct orion_addr_map_cfg *cfg,const struct orion_addr_map_info *info);
Signature · 1 line
1.7.9.5
PATCH 13/32 arm: mach-dove: use ORION_ADDR_MAP_NO_REMAP
In reply to Thomas PetazzoniCurrently, the addr_map_info structure uses an "int remap" to allow SoC-specific code to specify whether a given address decoding window should be remapped, and if so, the remapping address. To do so, it tests if the int is less than zero (no remap) or greater than zero. Unfortunately, this prevents any remapping address that is higher than 2 GB.
In this commit, we convert mach-dove addr_map_info definition to use ORION_ADDR_MAP_NO_REMAP.
--- arch/arm/mach-dove/addr-map.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/arm/mach-dove/addr-map.c +5 -5
@@ -61,28 +61,28 @@ static const struct __initdata orion_addr_map_info addr_map_info[] = {TARGET_PCIE1 ,ATTR_PCIE_IO ,DOVE_PCIE1_IO_BUS_BASE }, { 2, DOVE_PCIE0_MEM_PHYS_BASE, DOVE_PCIE0_MEM_SIZE, -TARGET_PCIE0 ,ATTR_PCIE_MEM ,-1 +TARGET_PCIE0 ,ATTR_PCIE_MEM ,ORION_ADDR_MAP_NO_REMAP , }, { 3, DOVE_PCIE1_MEM_PHYS_BASE, DOVE_PCIE1_MEM_SIZE, -TARGET_PCIE1 ,ATTR_PCIE_MEM ,-1 +TARGET_PCIE1 ,ATTR_PCIE_MEM ,ORION_ADDR_MAP_NO_REMAP , }, /* * Window for CESA engine. */ { 4, DOVE_CESA_PHYS_BASE, DOVE_CESA_SIZE, -TARGET_CESA ,ATTR_CESA ,-1 +TARGET_CESA ,ATTR_CESA ,ORION_ADDR_MAP_NO_REMAP , }, /* * Window to the BootROM for Standby and Sleep Resume */ { 5, DOVE_BOOTROM_PHYS_BASE, DOVE_BOOTROM_SIZE, -TARGET_BOOTROM ,ATTR_BOOTROM ,-1 +TARGET_BOOTROM ,ATTR_BOOTROM ,ORION_ADDR_MAP_NO_REMAP , }, /* * Window to the PMU Scratch Pad space */ { 6, DOVE_SCRATCHPAD_PHYS_BASE, DOVE_SCRATCHPAD_SIZE, -TARGET_SCRATCHPAD ,ATTR_SCRATCHPAD ,-1 +TARGET_SCRATCHPAD ,ATTR_SCRATCHPAD ,ORION_ADDR_MAP_NO_REMAP , }, /* End marker */ { -1, 0, 0, 0, 0, 0 }
Signature · 1 line
1.7.9.5
PATCH 14/32 arm: mach-kirkwood: use ORION_ADDR_MAP_NO_REMAP
In reply to Thomas PetazzoniCurrently, the addr_map_info structure uses an "int remap" to allow SoC-specific code to specify whether a given address decoding window should be remapped, and if so, the remapping address. To do so, it tests if the int is less than zero (no remap) or greater than zero. Unfortunately, this prevents any remapping address that is higher than 2 GB.
In this commit, we convert mach-kirkwood addr_map_info definition to use ORION_ADDR_MAP_NO_REMAP.
--- arch/arm/mach-kirkwood/addr-map.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm/mach-kirkwood/addr-map.c +2 -2
@@ -64,13 +64,13 @@ static const struct __initdata orion_addr_map_info addr_map_info[] = {* Window for NAND controller. */ {4 ,KIRKWOOD_NAND_MEM_PHYS_BASE ,KIRKWOOD_NAND_MEM_SIZE , -TARGET_DEV_BUS ,ATTR_DEV_NAND ,-1 +TARGET_DEV_BUS ,ATTR_DEV_NAND ,ORION_ADDR_MAP_NO_REMAP , }, /* * Window for SRAM. */ { 5, KIRKWOOD_SRAM_PHYS_BASE, KIRKWOOD_SRAM_SIZE, -TARGET_SRAM ,ATTR_SRAM ,-1 +TARGET_SRAM ,ATTR_SRAM ,ORION_ADDR_MAP_NO_REMAP , }, /* End marker */ { -1, 0, 0, 0, 0, 0 }
Signature · 1 line
1.7.9.5
PATCH 15/32 arm: mach-mvebu: use ORION_ADDR_MAP_NO_REMAP
In reply to Thomas PetazzoniCurrently, the addr_map_info structure uses an "int remap" to allow SoC-specific code to specify whether a given address decoding window should be remapped, and if so, the remapping address. To do so, it tests if the int is less than zero (no remap) or greater than zero. Unfortunately, this prevents any remapping address that is higher than 2 GB.
In this commit, we convert mach-mvebu addr_map_info definition to use ORION_ADDR_MAP_NO_REMAP.
--- arch/arm/mach-mvebu/addr-map.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/mach-mvebu/addr-map.c +1 -1
@@ -42,7 +42,7 @@ armada_xp_addr_map_info[] = {* Window for the BootROM, needed for SMP on Armada XP */ {0 ,0xfff00000 ,SZ_1M ,ARMADA_XP_TARGET_DEV_BUS , - ARMADA_XP_ATTR_DEV_BOOTROM, -1 }, + ARMADA_XP_ATTR_DEV_BOOTROM, ORION_ADDR_MAP_NO_REMAP },/* End marker */ { -1, 0, 0, 0, 0, 0 }, };
Signature · 1 line
1.7.9.5
PATCH 16/32 arm: mach-orion5x: use ORION_ADDR_MAP_NO_REMAP
In reply to Thomas PetazzoniCurrently, the addr_map_info structure uses an "int remap" to allow SoC-specific code to specify whether a given address decoding window should be remapped, and if so, the remapping address. To do so, it tests if the int is less than zero (no remap) or greater than zero. Unfortunately, this prevents any remapping address that is higher than 2 GB.
In this commit, we convert mach-orion5x addr_map_info definition to use ORION_ADDR_MAP_NO_REMAP.
--- arch/arm/mach-orion5x/addr-map.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm/mach-orion5x/addr-map.c +2 -2
@@ -93,10 +93,10 @@ static const struct __initdata orion_addr_map_info addr_map_info[] = {TARGET_PCI ,ATTR_PCI_IO ,ORION5X_PCI_IO_BUS_BASE }, { 2, ORION5X_PCIE_MEM_PHYS_BASE, ORION5X_PCIE_MEM_SIZE, -TARGET_PCIE ,ATTR_PCIE_MEM ,-1 +TARGET_PCIE ,ATTR_PCIE_MEM ,ORION_ADDR_MAP_NO_REMAP , }, { 3, ORION5X_PCI_MEM_PHYS_BASE, ORION5X_PCI_MEM_SIZE, -TARGET_PCI ,ATTR_PCI_MEM ,-1 +TARGET_PCI ,ATTR_PCI_MEM ,ORION_ADDR_MAP_NO_REMAP , }, /* End marker */ { -1, 0, 0, 0, 0, 0 }
Signature · 1 line
1.7.9.5
PATCH 17/32 arm: plat-orion: convert 'int remap' to 'u32 remap'
In reply to Thomas PetazzoniNow that we have ORION_ADDR_MAP_NO_REMAP in place, we can easily convert the 'int remap' field to a 'u32 remap' field, and accept any remapping address, except the magical ORION_ADDR_MAP_NO_REMAP.
--- arch/arm/plat-orion/addr-map.c | 4 ++-- arch/arm/plat-orion/include/plat/addr-map.h | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/arm/plat-orion/addr-map.c +2 -2
@@ -69,7 +69,7 @@ static int __init orion_cpu_win_can_remap(const struct orion_addr_map_cfg *cfg,void __init orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg,const int win,const u32 base,const u32 size,const u8 target, -const u8 attr,const int remap) +const u8 attr,const u32 remap) {void __iomem *addr = cfg->win_cfg_base (cfg, win);u32 ctrl, base_high, remap_addr;
@@ -86,7 +86,7 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg,writel (base_high, addr +WIN_BASE_OFF );writel (ctrl, addr +WIN_CTRL_OFF );if (cfg->cpu_win_can_remap (cfg, win)) { -if (remap <0 ) +if (remap ==ORION_ADDR_MAP_NO_REMAP ) remap_addr = base;else remap_addr = remap;
arch/arm/plat-orion/include/plat/addr-map.h +3 -3
@@ -38,10 +38,10 @@ struct orion_addr_map_info {const u32 size;const u8 target;const u8 attr; -const int remap; +const u32 remap; }; -#define ORION_ADDR_MAP_NO_REMAP (-1) +#define ORION_ADDR_MAP_NO_REMAP (0xFFFFFFFF)void __init orion_config_wins (struct orion_addr_map_cfg *cfg,const struct orion_addr_map_info *info);
@@ -49,7 +49,7 @@ void __init orion_config_wins(struct orion_addr_map_cfg *cfg,void __init orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg,const int win,const u32 base,const u32 size,const u8 target, -const u8 attr,const int remap); +const u8 attr,const u32 remap);void __init orion_setup_cpu_mbus_target (const struct orion_addr_map_cfg *cfg,const void __iomem *ddr_window_cpu_base);
Signature · 1 line
1.7.9.5
PATCH 18/32 arm: plat-orion: remove __init from addr-map functions needed after boot time
In reply to Thomas PetazzoniThe I/O and memory windows of PCIe interfaces will now be configurable dynamically, while the system runs, so we need to keep some the plat-orion/addr-map.c functions around even after the system initialization has completed.
--- arch/arm/plat-orion/addr-map.c | 18 +++++++++--------- arch/arm/plat-orion/include/plat/addr-map.h | 8 ++++---- 2 files changed, 13 insertions(+), 13 deletions(-)
arch/arm/plat-orion/addr-map.c +9 -9
@@ -48,7 +48,7 @@ EXPORT_SYMBOL_GPL(mv_mbus_dram_info);/* * Default implementation */ -static void __init __iomem * +static void __iomem *orion_win_cfg_base (const struct orion_addr_map_cfg *cfg,int win) {return cfg->bridge_virt_base + (win <<4 );
@@ -57,8 +57,8 @@ orion_win_cfg_base(const struct orion_addr_map_cfg *cfg, int win)/* * Default implementation */ -static int __init orion_cpu_win_can_remap (const struct orion_addr_map_cfg *cfg, -const int win) +static int orion_cpu_win_can_remap (const struct orion_addr_map_cfg *cfg, +const int win) {if (win < cfg->remappable_wins)return 1 ;
@@ -66,10 +66,10 @@ static int __init orion_cpu_win_can_remap(const struct orion_addr_map_cfg *cfg,return 0 ; } -void __init orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg, -const int win,const u32 base, -const u32 size,const u8 target, -const u8 attr,const u32 remap) +void orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg, +const int win,const u32 base, +const u32 size,const u8 target, +const u8 attr,const u32 remap) {void __iomem *addr = cfg->win_cfg_base (cfg, win);u32 ctrl, base_high, remap_addr;
@@ -95,8 +95,8 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, } } -static void __init orion_disable_cpu_win (const struct orion_addr_map_cfg *cfg, -const int win) +static void orion_disable_cpu_win (const struct orion_addr_map_cfg *cfg, +const int win) {void __iomem *addr = cfg->win_cfg_base (cfg, win);
arch/arm/plat-orion/include/plat/addr-map.h +4 -4
@@ -46,10 +46,10 @@ struct orion_addr_map_info {void __init orion_config_wins (struct orion_addr_map_cfg *cfg,const struct orion_addr_map_info *info); -void __init orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg, -const int win,const u32 base, -const u32 size,const u8 target, -const u8 attr,const u32 remap); +void orion_setup_cpu_win (const struct orion_addr_map_cfg *cfg, +const int win,const u32 base, +const u32 size,const u8 target, +const u8 attr,const u32 remap);void __init orion_setup_cpu_mbus_target (const struct orion_addr_map_cfg *cfg,const void __iomem *ddr_window_cpu_base);
Signature · 1 line
1.7.9.5
PATCH 19/32 arm: plat-orion: introduce orion_{alloc, free}_cpu_win() functions
In reply to Thomas PetazzoniIn the address decoding code, we implement two new functions:orion_alloc_cpu_win() and orion_free_cpu_win(). The first function finds an unused address decoding window, and configures it according to the given arguments (in terms of base address, size, target, attributes). The second function frees an address decoding window, given a physical base address.
Those two new functions will be used by the PCIe code, which needs to dynamically register address decoding windows depending on the PCIe devices that are detected.
The orion_free_cpu_win() function is only here to handle error cases in the PCIe devices initialization, in the normal case, address decoding windows are never freed.
--- arch/arm/plat-orion/addr-map.c | 75 +++++++++++++++++++++++++++ arch/arm/plat-orion/include/plat/addr-map.h | 7 +++ 2 files changed, 82 insertions(+)
arch/arm/plat-orion/addr-map.c +75 -0
@@ -108,6 +108,81 @@ static void orion_disable_cpu_win(const struct orion_addr_map_cfg *cfg, } } +static int orion_window_is_free (const struct orion_addr_map_cfg *cfg, +const int win) +{ +void __iomem *addr = cfg->win_cfg_base (cfg, win); +u32 ctrl =readl (addr +WIN_CTRL_OFF ); +return !(ctrl &WIN_CTRL_ENABLE ); +} + +/* +* Find an unused address decoding window, and enable it according to +* the arguments passed (base, size, target, attributes, remap). +*/ +int orion_alloc_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base,const u32 size, +const u8 target,const u8 attr,const u32 remap) +{ +int win; + +/* +* If we don't need to remap, try to first allocate a window +* that doesn't have remapping capabilities, since those are +* pretty limited in number. +*/ +if (remap ==ORION_ADDR_MAP_NO_REMAP ) { +for (win = cfg->remappable_wins; win < cfg->num_wins; win++) +if (orion_window_is_free (cfg, win)) +break ; + +if (win < cfg->num_wins) { +orion_setup_cpu_win (cfg, win, base, size, target, attr, remap); +return 0 ; + } + } + +/* +* We need a remap window, or we haven't found a non-remap +* capable window. +*/ +for (win =0 ; win < cfg->num_wins; win++) +if (orion_window_is_free (cfg, win)) +break ; + +if (win < cfg->num_wins) { +orion_setup_cpu_win (cfg, win, base, size, target, attr, remap); +return 0 ; + } + +/* No window found */ +return -ENOMEM ; +} + +/* +* Free an address decoding window, given its base address. +*/ +int orion_free_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base) +{ +int win; + +for (win =0 ; win < cfg->num_wins; win++) { +void __iomem *addr = cfg->win_cfg_base (cfg, win); +u32 winbase =readl (addr +WIN_BASE_OFF ); + +if (orion_window_is_free (cfg, win)) +continue ; + +if (winbase == (base &0xffff0000 )) { +orion_disable_cpu_win (cfg, win); +return 0 ; + } + } + +return -EINVAL ; +} +/* * Configure a number of windows. */
arch/arm/plat-orion/include/plat/addr-map.h +7 -0
@@ -51,6 +51,13 @@ void orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, const u32 size, const u8 target, const u8 attr, const u32 remap); +int orion_alloc_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base,const u32 size, +const u8 target,const u8 attr,const u32 remap); + +int orion_free_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base); +void __init orion_setup_cpu_mbus_target (const struct orion_addr_map_cfg *cfg,const void __iomem *ddr_window_cpu_base);#endif
Signature · 1 line
1.7.9.5
PATCH 20/32 arm: plat-orion: remove __init from PCIe functions needed after boot time
In reply to Thomas PetazzoniThe orion_pcie_set_local_bus_nr() function will be used by the Marvell PCIe driver in its PCI-to-PCI bridge emulation. This means that this function can now get called at any point during the system execution, and no longer at boot time. For example, if PCIe devices appear/disappear during the system execution (either through hotplug, or manipulation of PCIe devices using /sys).
--- arch/arm/plat-orion/pcie.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/plat-orion/pcie.c +1 -1
@@ -79,7 +79,7 @@ int orion_pcie_get_local_bus_nr(void __iomem *base)return (stat >>PCIE_STAT_BUS_OFFS ) &PCIE_STAT_BUS_MASK ; } -void __init orion_pcie_set_local_bus_nr (void __iomem *base,int nr) +void orion_pcie_set_local_bus_nr (void __iomem *base,int nr) {u32 stat;
Signature · 1 line
1.7.9.5
PATCH 21/32 arm: mvebu: add functions to alloc/free PCIe decoding windows
In reply to Thomas PetazzoniThis commit adds two functions armada_370_xp_alloc_pcie_window() and armada_370_xp_free_pcie_window() that respectively allocate and free an address decoding window pointing to either a memory or I/O region of a PCIe device.
Those functions will be used by the PCIe driver to create and remove those regions depending on the PCIe devices that are detected.
--- arch/arm/mach-mvebu/addr-map.c | 87 +++++++++++++++++++++++++-- arch/arm/mach-mvebu/common.h | 1 + arch/arm/mach-mvebu/include/mach/addr-map.h | 10 +++ 3 files changed, 94 insertions(+), 4 deletions(-) create mode 100644 arch/arm/mach-mvebu/include/mach/addr-map.h
arch/arm/mach-mvebu/addr-map.c +83 -4
@@ -24,14 +24,10 @@#define ARMADA_XP_TARGET_DEV_BUS 1#define ARMADA_XP_ATTR_DEV_BOOTROM 0x1D#define ARMADA_XP_TARGET_ETH1 3 -#define ARMADA_XP_TARGET_PCIE_0_2 4#define ARMADA_XP_TARGET_ETH0 7 -#define ARMADA_XP_TARGET_PCIE_1_3 8#define ARMADA_370_TARGET_DEV_BUS 1#define ARMADA_370_ATTR_DEV_BOOTROM 0x1D -#define ARMADA_370_TARGET_PCIE_0 4 -#define ARMADA_370_TARGET_PCIE_1 8#define ARMADA_WINDOW_8_PLUS_OFFSET 0x90#define ARMADA_SDRAM_ADDR_DECODING_OFFSET 0x180
@@ -89,6 +85,89 @@ static struct __initdata orion_addr_map_cfg addr_map_cfg = { .win_cfg_base = armada_cfg_base, }; +#ifdef CONFIG_PCI +/* +* This structure and the following arrays allow to map a PCIe (port, +* lane) tuple to the corresponding (target, attribute) tuple needed +* to configure an address decoding window for the given PCIe (port, +* lane). +*/ +struct pcie_mapping { +int port; +int lane; +u8 target; +u8 attr; +}; + +struct pcie_mapping armada_xp_pcie_mappings[] = { + { .port =0 , .lane =0 , .target =4 , .attr =0xE0 }, + { .port =0 , .lane =1 , .target =4 , .attr =0xD0 }, + { .port =0 , .lane =2 , .target =4 , .attr =0xB0 }, + { .port =0 , .lane =3 , .target =4 , .attr =0x70 }, + { .port =1 , .lane =0 , .target =8 , .attr =0xE0 }, + { .port =1 , .lane =1 , .target =8 , .attr =0xD0 }, + { .port =1 , .lane =2 , .target =8 , .attr =0xB0 }, + { .port =1 , .lane =3 , .target =8 , .attr =0x70 }, + { .port =2 , .lane =0 , .target =4 , .attr =0xF0 }, + { .port =3 , .lane =0 , .target =8 , .attr =0xF0 }, + { .port =-1 }, +}; + +struct pcie_mapping armada_370_pcie_mappings[] = { + { .port =0 , .lane =0 , .target =4 , .attr =0xE0 }, + { .port =1 , .lane =0 , .target =8 , .attr =0xE0 }, + { .port =-1 }, +}; + +/* +* This function sets up a new address decoding window at the given +* base address, pointing to the given PCIe interface (through +* pcie_port and pcie_lane). +*/ +int armada_370_xp_alloc_pcie_window (int pcie_port,int pcie_lane, +u32 base,u32 remap,u32 size, +int type) +{ +struct pcie_mapping *mapping, *mappings; +u8 target, attr; + +if (of_machine_is_compatible ("marvell,armadaxp" )) + mappings = armada_xp_pcie_mappings; +else if (of_machine_is_compatible ("marvell,armada370" )) + mappings = armada_370_pcie_mappings; +else +return -ENODEV ; + +for (mapping = mappings; mapping->port !=-1 ; mapping++) +if (mapping->port == pcie_port && mapping->lane == pcie_lane) +break ; + +if (mapping->port ==-1 ) +return -ENODEV ; + + target = mapping->target; + attr = mapping->attr; + +/* +* Bit 3 of the attributes indicates that it is a +* memory region, as opposed to an I/O region +*/ +if (type ==IORESOURCE_MEM ) + attr |= (1 <<3 ); + +return orion_alloc_cpu_win (&addr_map_cfg, base, size, target, attr, remap); +} + +/* +* Frees an address decoding window previously set up by +* armada_370_xp_setup_pcie_window(). +*/ +int armada_370_xp_free_pcie_window (u32 base) +{ +return orion_free_cpu_win (&addr_map_cfg, base); +} +#endif +static int __init armada_setup_cpu_mbus (void ) {struct device_node *np;
arch/arm/mach-mvebu/common.h +1 -0
@@ -25,4 +25,5 @@ int armada_370_xp_coherency_init(void);int armada_370_xp_pmsu_init (void );void armada_xp_secondary_startup (void );extern struct smp_operations armada_xp_smp_ops; +#endif
arch/arm/mach-mvebu/include/mach/addr-map.h new file +10 -0
@@ -0,0 +1,10 @@ +#ifndef MVEBU_ADDR_MAP_H +#define MVEBU_ADDR_MAP_H + +#include <plat/addr-map.h> + +int armada_370_xp_alloc_pcie_window (int pcie_port,int pcie_lane, +u32 base,u32 remap,u32 size,int type); +int armada_370_xp_free_pcie_window (unsigned long base); + +#endif /* MVEBU_ADDR_MAP_H */
Signature · 1 line
1.7.9.5
PATCH 22/32 arm: plat-orion: make common PCIe code usable on mvebu
In reply to Thomas Petazzonimvebu is a new-style Orion platform, so it only selects PLAT_ORION, but not PLAT_ORION_LEGACY. It will however need the common PCIe code from plat-orion, so make this code available for PLAT_ORION platforms as a whole, and not only PLAT_ORION_LEGACY platforms.
We also take this opportunity to build the PCIe code only when CONFIG_PCI is enabled.
--- arch/arm/plat-orion/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm/plat-orion/Makefile +2 -1
@@ -4,7 +4,8 @@ ccflags-$(CONFIG_ARCH_MULTIPLATFORM) := -I$(srctree)/$(src)/include obj-y += addr-map.o +obj-$(CONFIG_PCI) += pcie.o orion-gpio-$(CONFIG_GENERIC_GPIO) += gpio.o -obj-$(CONFIG_PLAT_ORION_LEGACY) += irq.o pcie.o time.o common.o mpp.o +obj-$(CONFIG_PLAT_ORION_LEGACY) += irq.o time.o common.o mpp.o obj-$(CONFIG_PLAT_ORION_LEGACY) += $(orion-gpio-y)
Signature · 1 line
1.7.9.5
PATCH 23/32 pci: infrastructure to add drivers in drivers/pci/host
In reply to Thomas PetazzoniAs agreed by the community, PCI host drivers will now be stored in drivers/pci/host. This commit adds this directory and the related Kconfig/Makefile changes to allow new drivers to be added in this directory.
--- drivers/pci/Kconfig | 2 ++ drivers/pci/Makefile | 3 +++ drivers/pci/host/Kconfig | 4 ++++ 3 files changed, 9 insertions(+) create mode 100644 drivers/pci/host/Kconfig
drivers/pci/Kconfig +2 -0
@@ -119,3 +119,5 @@ config PCI_IOAPIC config PCI_LABEL def_bool y if (DMI || ACPI) select NLS + +source "drivers/pci/host/Kconfig"
drivers/pci/Makefile +3 -0
@@ -67,3 +67,6 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_OF) += of.o ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG + +# PCI host controller drivers +obj-y += host/
drivers/pci/host/Kconfig new file +4 -0
@@ -0,0 +1,4 @@ +menu "PCI host controller drivers" + depends on PCI + +endmenu
Signature · 1 line
1.7.9.5
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniThis driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the hw_pci operations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and the pci_ops operations that are used by the PCI core to read/write the configuration space of PCI devices.
Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
In addition, this driver enumerates the different PCIe slots, and for those having a device plugged in, it sets up the necessary address decoding windows, using the new armada_370_xp_alloc_pcie_window() function from mach-mvebu/addr-map.c.
--- .../devicetree/bindings/pci/mvebu-pci.txt | 174 +++++ drivers/pci/host/Kconfig | 6 + drivers/pci/host/Makefile | 4 + drivers/pci/host/pci-mvebu.c | 688 ++++++++++++++++++++ 4 files changed, 872 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/mvebu-pci.txt create mode 100644 drivers/pci/host/Makefile create mode 100644 drivers/pci/host/pci-mvebu.c
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +174 -0
@@ -0,0 +1,174 @@ +* Marvell EBU PCIe interfaces + +Mandatory properties: +- compatible: for now only "marvell,armada-370-xp-pcie" is supported +- #address-cells, set to <3> +- #size-cells, set to <2> +- #interrupt-cells, set to <1> +- bus-range: PCI bus numbers covered +- ranges: standard PCI-style address ranges, describing the PCIe + registers for each PCIe interface, and then ranges for the PCI + memory and I/O regions. +- interrupt-map-mask and interrupt-map are standard PCI Device Tree + properties to describe the interrupts associated to each PCI + interface. + +In addition, the Device Tree node must have sub-nodes describing each +PCIe interface, having the following mandatory properties: +- reg: the address and size of the PCIe registers (translated + addresses according to the ranges property of the parent) +- clocks: the clock associated to this PCIe interface +- marvell,pcie-port: the physical PCIe port number +- status: either "disabled" or "okay" + +and the following optional properties: +- marvell,pcie-lane: the physical PCIe lane number, for ports having + multiple lanes. If this property is not found, we assume that the + value is 0. + +Example: + +pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */ + 0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */ + 0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */ + 0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>; + interrupt-map = <0x0800 0 0 1 &mpic 58 + 0x1000 0 0 1 &mpic 59 + 0x1800 0 0 1 &mpic 60 + 0x2000 0 0 1 &mpic 61 + 0x2800 0 0 1 &mpic 62 + 0x3000 0 0 1 &mpic 63 + 0x3800 0 0 1 &mpic 64 + 0x4000 0 0 1 &mpic 65 + 0x4800 0 0 1 &mpic 99 + 0x5000 0 0 1 &mpic 103>; + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 0,1 { + device_type = "pciex"; + reg = <0x1000 0 0xd0044000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 6>; + status = "disabled"; + }; + + pcie at 0,2 { + device_type = "pciex"; + reg = <0x1800 0 0xd0048000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 7>; + status = "disabled"; + }; + + pcie at 0,3 { + device_type = "pciex"; + reg = <0x2000 0 0xd004C000 0 0xC000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 8>; + status = "disabled"; + }; + + pcie at 1,0 { + device_type = "pciex"; + reg = <0x2800 0 0xd0080000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 9>; + status = "disabled"; + }; + + pcie at 1,1 { + device_type = "pciex"; + reg = <0x3000 0 0xd0084000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 10>; + status = "disabled"; + }; + + pcie at 1,2 { + device_type = "pciex"; + reg = <0x3800 0 0xd0088000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 11>; + status = "disabled"; + }; + + pcie at 1,3 { + device_type = "pciex"; + reg = <0x4000 0 0xd008C000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 12>; + status = "disabled"; + }; + pcie at 2,0 { + device_type = "pciex"; + reg = <0x4800 0 0xd0042000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <2>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 26>; + status = "disabled"; + }; + + pcie at 3,0 { + device_type = "pciex"; + reg = <0x5000 0 0xd0082000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <3>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 27>; + status = "disabled"; + }; +};
drivers/pci/host/Kconfig +6 -0
@@ -1,4 +1,10 @@ menu "PCI host controller drivers" depends on PCI +config PCI_MVEBU + bool "Marvell EBU PCIe controller" + depends on ARCH_MVEBU + select PCI_SW_HOST_BRIDGE + select PCI_SW_PCI_PCI_BRIDGE + endmenu
drivers/pci/host/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +CFLAGS_pci-mvebu.o += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/include
drivers/pci/host/pci-mvebu.c new file +688 -0
688 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
drivers/pci/host/Makefile line 4
Quoted 10 lines · Thomas Petazzoni
drivers/pci/host/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +CFLAGS_pci-mvebu.o += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/include
This does not seem like a good idea to me. We should not include architecture specific directories from a driver directory.
What are the header files you need here?
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile new file +7
-0+/* + * This product ID is registered by Marvell, and used when the Marvell + * SoC is not the root complex, but an endpoint on the PCIe bus. It is + * therefore safe to re-use this PCI ID for our emulated PCI-to-PCI + * bridge. + */ +#define MARVELL_EMULATED_PCI_PCI_BRIDGE_ID 0x7846
Just a side note: What happens if you have two of these systems and connect them over PCIe, putting one of them into host mode and the other into endpoint mode?
drivers/pci/host/Makefile
Quoted 12 lines
drivers/pci/host/Makefile new file +12
-0+static void mvebu_pcie_setup_io_window(struct mvebu_pcie_port *port, + int enable) +{ + unsigned long iobase, iolimit; + + if (port->bridge.iolimit < port->bridge.iobase) + return; + + iolimit = 0xFFF | ((port->bridge.iolimit & 0xF0) << 8) | + (port->bridge.iolimitupper << 16); + iobase = ((port->bridge.iobase & 0xF0) << 8) | + (port->bridge.iobaseupper << 16);
I don't understand this code with the masks and shifts. Could you add a comment here for readers like me?
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile new file +7
-0+ +/* + * Initialize the configuration space of the PCI-to-PCI bridge + * associated with the given PCIe interface. + */ +static void mvebu_sw_pci_bridge_init(struct mvebu_pcie_port *port) +{
As mentioned, I'm still skeptical of the sw_pci_bridge approach, so I'm not commenting on the details of your implementations (they seem fine on a first look though)
drivers/pci/host/Makefile
Quoted 9 lines
drivers/pci/host/Makefile new file +9
-0+ /* Get the I/O and memory ranges from DT */ + while ((range = of_pci_process_ranges(np, &res, range)) != NULL) { + if (resource_type(&res) == IORESOURCE_IO) { + memcpy(&pcie->io, &res, sizeof(res)); + memcpy(&pcie->realio, &res, sizeof(res)); + pcie->io.name = "I/O"; + pcie->realio.start &= 0xFFFFF; + pcie->realio.end &= 0xFFFFF; + }
The bit masking seems fishy here. What exactly are you doing, does this just assume you have a 1MB window at most?
Maybe something like
pcie->realio.start = 0; pcie->realio.end = pcie->io.end - pcie->io.start;
I suppose you also need to fix up pcie->io to be in IORESOURCE_MEM space instead of IORESOURCE_IO, or fix the of_pci_process_ranges function to return it in a different way.
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile new file +7
-0+static int mvebu_pcie_init(void) +{ + return platform_driver_probe(&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall(mvebu_pcie_init);
You don't have to do it, but I wonder if this could be a module with unload support instead.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Arnd Bergmann,
Quoted 15 lines · Arnd Bergmann
drivers/pci/host/Makefile line 4
Quoted 10 lines · Thomas Petazzoni
drivers/pci/host/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +CFLAGS_pci-mvebu.o += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/includeThis does not seem like a good idea to me. We should not include architecture specific directories from a driver directory.
Quoted 3 lines
What are the header files you need here?
From the patch itself:
+#include <plat/pcie.h>
+#include <mach/addr-map.h>
<plat/pcie.h> is needed for a few PCIe functions shared with earlier families of Marvell SoC. My plan is that once this PCI driver gets accepted, I work on migrating the earlier Marvell SoC families to using this PCI driver, and therefore those functions would ultimately move in the driver in drivers/pci/host/, which would remove the <plat/pcie.h>.
The <mach/addr-map.h> is here to access the address decoding windows allocation/free API. And for this, there is no other long term plan than having an API provided by the platform code in arch/arm/, and used by drivers. Some other drivers may have to use this API as well in the future.
I think that completely preventing <mach/> and <plat/> includes from drivers is not possible. Some sub-architectures will also have some bizarre mechanism to handle (in our case the address decoding windows), for which there is no kernel-wide API and kernel-wide subsystem to handle it. In such cases, a sub-architecture specific solution is really the only reasonable way, and in this case, we have to include the sub-architecture headers.
Note that I have been careful to use CFLAGS_pci-mvebu.o, so that those include paths only apply to *this* driver. I added a separate dummy driver in drivers/pci/host/, and verified that those include paths are not used when building this other driver. So those special CFLAGS are still compatible with the multiplatform kernel.
Quoted 5 lines
Quoted 5 lines
+/*
+ * This product ID is registered by Marvell, and used when the Marvell
+ * SoC is not the root complex, but an endpoint on the PCIe bus. It is
Quoted 8 lines
Quoted 4 lines
+ * therefore safe to re-use this PCI ID for our emulated PCI-to-PCI
+ * bridge.
+ */
+#defineMARVELL_EMULATED_PCI_PCI_BRIDGE_ID0x7846Just a side note: What happens if you have two of these systems and connect them over PCIe, putting one of them into host mode and the other into endpoint mode?
I am not a PCI expert, but I don't think it would cause issues. Maybe Jason Gunthorpe can comment on this, as he originally suggested to re-use this PCI ID.
Quoted 8 lines
Quoted 8 lines
+static void mvebu_pcie_setup_io_window(struct
mvebu_pcie_port*port,drivers/pci/host/Makefile new file +6
-0+ int enable) +{ + unsigned long iobase, iolimit; + + if (port->bridge.iolimit < port->bridge.iobase) + return;
Quoted 8 lines
drivers/pci/host/Makefile
Quoted 5 lines
drivers/pci/host/Makefile new file +5
-0+ + iolimit = 0xFFF | ((port->bridge.iolimit & 0xF0) << 8) | + (port->bridge.iolimitupper << 16); + iobase = ((port->bridge.iobase & 0xF0) << 8) | + (port->bridge.iobaseupper << 16);I don't understand this code with the masks and shifts. Could you add a comment here for readers like me?
Sure, will do.
It basically comes from the PCI-to-PCI bridge specification, which explains how the I/O address and I/O limit is split into two 16 bits registers, with those bizarre shifts and hardcoded values. I'll put a reference to the relevant section of the PCI-to-PCI bridge specification here.
Quoted 11 lines
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile new file +7
-0+ +/* + * Initialize the configuration space of the PCI-to-PCI bridge + * associated with the given PCIe interface. + */ +static void mvebu_sw_pci_bridge_init(struct mvebu_pcie_port *port) +{As mentioned, I'm still skeptical of the
sw_pci_bridgeapproach, so I'm not commenting on the details of your implementations (they seem fine on a first look though)
Yes, I understood your were still skeptical. But as I've mentioned in other e-mails, I still haven't seen any other serious alternate proposal that takes into account the need for dynamic assignment of addresses.
Quoted 5 lines
Quoted 5 lines
drivers/pci/host/Makefile new file +2
-0+ /* Get the I/O and memory ranges from DT */ + while ((range = of_pci_process_ranges(np, &res, range)) !=NULL) {
drivers/pci/host/Makefile new file +2
-0+ if (resource_type(&res) == IORESOURCE_IO) { + memcpy(&pcie->io, &res, sizeof(res));
Quoted 8 lines
drivers/pci/host/Makefile
Quoted 5 lines
drivers/pci/host/Makefile new file +5
-0+ memcpy(&pcie->realio, &res, sizeof(res)); + pcie->io.name = "I/O"; + pcie->realio.start &= 0xFFFFF; + pcie->realio.end &= 0xFFFFF; + }The bit masking seems fishy here. What exactly are you doing, does this just assume you have a 1MB window at most?
Basically, I have two resources for the I/O:
* One described in the DT, from 0xC0000000 to 0xC00FFFFF which will be used to create the address decoding windows for the I/O regions of the different PCIe interfaces. The PCI I/O virtual address 0xffe00000 will be mapped to those physical addresses. Those address decoding windows are configured with the special "remap" mechanism that ensures that if an access is made at 0xC0000000 + offset, it will appear on the PCI bus as an I/O access at address "offset".
* One covering the low addresses 0x0 -> 0xFFFFF (pcie->realio), which is used to tell the Linux PCI subsystem from which address range it should assign I/O addresses.
Quoted 4 lines
Maybe something like
pcie->realio.start = 0; pcie->realio.end = pcie->io.end - pcie->io.start;
Indeed, that would result in the same values. If you find it clearer, I'm fine with it.
Quoted 3 lines
I suppose you also need to fix up
pcie->ioto be inIORESOURCE_MEMspace instead ofIORESOURCE_IO, or fix theof_pci_process_rangesfunction to return it in a different way.
Ok.
Quoted 10 lines
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile new file +7
-0+static int mvebu_pcie_init(void) +{ + return platform_driver_probe(&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall(mvebu_pcie_init);You don't have to do it, but I wonder if this could be a module with unload support instead.
This has already been discussed in the review of PATCHv2. Please see
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-January/145580.html.
Basically, doing a module_init() initialization fails, because the XHCI USB quirks are executed before we have the chance to create the address decoding windows, which crashes the kernel at boot time (and we have one platform where an USB 3.0 XHCI controller sits on the PCIe bus). Bjorn Helgaas has acknowledged the problem in
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-February/148292.html:
"""
This is not really a problem in your code; it's a generic PCI core problem. pci_scan_root_bus() does everything including creating the root bus, scanning it, and adding the devices we find. At the point where we add a device (pci_bus_add_device()), it should be ready for a driver to claim it -- all resource assignment should already be done.
I don't think it's completely trivial to fix this in the PCI core yet (but we're moving in that direction) because we have some boot-time ordering issues, e.g., x86 scans the root buses before we know about the address space consumed by ACPI devices, so we can't just assign the resources when we scan the bus.
"""
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Thomas Petazzoni
Quoted 5 lines
Quoted 5 lines
+/*
+ * This product ID is registered by Marvell, and used when the Marvell
+ * SoC is not the root complex, but an endpoint on the PCIe bus. It isQuoted 8 lines
Quoted 4 lines
+ * therefore safe to re-use this PCI ID for our emulated PCI-to-PCI
+ * bridge.
+ */
+#defineMARVELL_EMULATED_PCI_PCI_BRIDGE_ID0x7846Just a side note: What happens if you have two of these systems and connect them over PCIe, putting one of them into host mode and the other into endpoint mode?
Quoted 3 lines
I am not a PCI expert, but I don't think it would cause issues. Maybe Jason Gunthorpe can comment on this, as he originally suggested to re-use this PCI ID.
The answer is a bit complex..
- Yes the Marvell SOC, or related, can be used as an end point, and you could pair two SOCs together over PCI
- When in end point mode the SOC *probably* should have its ID reassigned. The ID is supposed to reflect the function of the device, not the HW vendor. In end point mode the ARM code controls how the device appears on PCIe.
So, if someone made an add-in card based around one of these Marvell SOCs then it still won't expose this ID to the kernel.
- Even if the ID is reused there is a PCI-PCI bridge version and a end device version, so drivers could still tell them apart by class matching.
FWIW, I would fetch the ID from the HW, as different SOC varients may vary the ID, though that is minor.
Basically, there is a theoretical problem here, but it is trivially solvable in two ways by whomever decides to actually do this.
Quoted 5 lines
Quoted 3 lines
I suppose you also need to fix up
pcie->ioto be inIORESOURCE_MEMspace instead ofIORESOURCE_IO, or fix theof_pci_process_rangesfunction to return it in a different way.Ok.
What does /proc/iomem say with your driver, are all the memory regions properly requested?
FWIW, the orion mbus window driver should also get an update to request memory regions, and IMHO, some of that code is better placed there than in the PCI driver. Specifically, I think it should:
- Request regions for the DDR ('System RAM') and internal regs
- Request regions for every mbus window that was programmed in, orion_addr_map_cfg - ideally updated to decode the target specifier.
- Request and unrequest regions for the PCI driver's dynamic requests.
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 23 lines · Thomas Petazzoni
Quoted 15 lines · Arnd Bergmann
drivers/pci/host/Makefile line 4
Quoted 10 lines · Thomas Petazzoni
drivers/pci/host/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +CFLAGS_pci-mvebu.o += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/includeThis does not seem like a good idea to me. We should not include architecture specific directories from a driver directory.
Quoted 1 line
What are the header files you need here?
From the patch itself:
drivers/pci/host/pci-mvebu.c new file +2
-0+#include <plat/pcie.h> +#include <mach/addr-map.h>
Quoted 5 lines
<
plat/pcie.h> is needed for a few PCIe functions shared with earlier families of Marvell SoC. My plan is that once this PCI driver gets accepted, I work on migrating the earlier Marvell SoC families to using this PCI driver, and therefore those functions would ultimately move in the driver indrivers/pci/host/, which would remove the <plat/pcie.h>.
Hmm, although it is a bit unusual, I would actually propose duplicating that code for now, and merging a copy into the new driver right away. This gets rid of the header file dependency and lets you just delete the old file when the other platforms are converted.
Quoted 5 lines
The <
mach/addr-map.h> is here to access the address decoding windows allocation/free API. And for this, there is no other long term plan than having an API provided by the platform code inarch/arm/, and used by drivers. Some other drivers may have to use this API as well in the future.
This is harder to do, but I'm sure we can find a solution. At least the addr-map.c code has no other dependencies than the plat/addr-map.h header, so we are fairly free to move it elsehwere.
I can think of several approaches that I'd prefer over your approach, although I have to admit that none of them makes me really happy:
a) arch/arm/common/mv-addr-map.c and arch/arm/include/asm/mach/mv-addr-map.h I assume that Russell will object to this one, but I let him speak for himself
b) drivers/bus/<something>
This would make a lot more sense if we followed the scheme I explained in my discussion to Jason Gunthorpe, where we basically treat this bus as a parent node in the device tree for anything that can get remapped. Without that change, it feels a little misplaced
c) drivers/misc/
I usually object to anything put in here and would certainly prefer the previous one over this.
Quoted 5 lines
Note that I have been careful to use CFLAGS_pci-mvebu.o, so that those include paths only apply to *this* driver. I added a separate dummy driver in
drivers/pci/host/, and verified that those include paths are not used when building this other driver. So those special CFLAGS are still compatible with the multiplatform kernel.
I understand this, but I think it would be better not to set a precedent for doing this.
Quoted 18 lines
Quoted 17 lines
Quoted 4 lines
+static void mvebu_pcie_setup_io_window(struct
mvebu_pcie_port*port,drivers/pci/host/Makefile new file +1
-0+ int enable)drivers/pci/host/Makefile
Quoted 10 lines
drivers/pci/host/Makefile new file +10
-0+{ + unsigned long iobase, iolimit; + + if (port->bridge.iolimit < port->bridge.iobase) + return; + + iolimit = 0xFFF | ((port->bridge.iolimit & 0xF0) << 8) | + (port->bridge.iolimitupper << 16); + iobase = ((port->bridge.iobase & 0xF0) << 8) | + (port->bridge.iobaseupper << 16);I don't understand this code with the masks and shifts. Could you add a comment here for readers like me?
Quoted 7 lines
Sure, will do.
It basically comes from the PCI-to-PCI bridge specification, which explains how the I/O address and I/O limit is split into two 16 bits registers, with those bizarre shifts and hardcoded values. I'll put a reference to the relevant section of the PCI-to-PCI bridge specification here.
Ok, I see. Thanks for the explanation.
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
drivers/pci/host/Makefile new file +2
-0+ /* Get the I/O and memory ranges from DT */ + while ((range = of_pci_process_ranges(np, &res, range)) !=NULL) {
drivers/pci/host/Makefile new file +2
-0+ if (resource_type(&res) == IORESOURCE_IO) { + memcpy(&pcie->io, &res, sizeof(res));Quoted 8 lines
drivers/pci/host/Makefile
Quoted 5 lines
drivers/pci/host/Makefile new file +5
-0+ memcpy(&pcie->realio, &res, sizeof(res)); + pcie->io.name = "I/O"; + pcie->realio.start &= 0xFFFFF; + pcie->realio.end &= 0xFFFFF; + }The bit masking seems fishy here. What exactly are you doing, does this just assume you have a 1MB window at most?
Basically, I have two resources for the I/O:
Quoted 7 lines
* One described in the DT, from
0xC0000000to0xC00FFFFFwhich will be used to create the address decoding windows for the I/O regions of the different PCIe interfaces. The PCI I/O virtual address0xffe00000will be mapped to those physical addresses. Those address decoding windows are configured with the special "remap" mechanism that ensures that if an access is made at0xC0000000+ offset, it will appear on the PCI bus as an I/O access at address "offset".
Right, I got this from reading the code. Unfortunately the of_pci_process_ranges functions from your earlier patch makes this a little confusing as it marks this resource as IORESOURCE_IO when in reality it is the IORESOURCE_MEM resource that describes where in MMIO space the I/O window is.
Quoted 11 lines
* One covering the low addresses
0x0->0xFFFFF(pcie->realio), which is used to tell the Linux PCI subsystem from which address range it should assign I/O addresses.Quoted 4 lines
Maybe something like
pcie->realio.start = 0; pcie->realio.end = pcie->io.end - pcie->io.start;Indeed, that would result in the same values. If you find it clearer, I'm fine with it.
Ideally we would read that from the ranges property as well, since it does not necessarily start at zero, although any other value would be a bit silly.
I definitely prefer the version I suggested over your version. On second thought I would make this
pcie->realio.start = PCIBIOS_MIN_IO; pcie->realio.end = min(pcie->io.end - pcie->io.start, IO_SPACE_LIMIT);
to ensure that we are inside of the limits.
Quoted 12 lines
Quoted 11 lines
drivers/pci/host/Makefile
Quoted 8 lines
drivers/pci/host/Makefile new file +7
-0+static int mvebu_pcie_init(void) +{ + return platform_driver_probe(&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall(mvebu_pcie_init);You don't have to do it, but I wonder if this could be a module with unload support instead.
Quoted 2 lines
This has already been discussed in the review of PATCHv2. Please see
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-January/145580.html.
Ok.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 5 lines · Arnd Bergmann
b) drivers/bus/<something>
This would make a lot more sense if we followed the scheme I explained in my discussion to Jason Gunthorpe, where we basically treat this bus as a parent node in the device tree for anything that can get remapped. Without that change, it feels a little misplaced
Also, FWIW, recall this related discussion and other possible DT binding:
http://www.spinics.net/lists/arm-kernel/msg219992.html
drivers/bus/orion-mbus.c feels like the right option, but when I looked at it, getting the DT binding, and full dynamicness setup seemed like it would be best done after non-DT support was purged, and that is somewhat contigent on getting the irqchip and timer stuff sorted (see my first attempt at that):
https://patchwork.kernel.org/patch/1852011/
Guess it depends where you want to draw the line on cleanups before something can be accepted..
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 19 lines · Jason Gunthorpe
Quoted 5 lines · Arnd Bergmann
b) drivers/bus/<something>
This would make a lot more sense if we followed the scheme I explained in my discussion to Jason Gunthorpe, where we basically treat this bus as a parent node in the device tree for anything that can get remapped. Without that change, it feels a little misplacedAlso, FWIW, recall this related discussion and other possible DT binding:
http://www.spinics.net/lists/arm-kernel/msg219992.html
drivers/bus/orion-mbus.cfeels like the right option, but when I looked at it, getting the DT binding, and full dynamicness setup seemed like it would be best done after non-DT support was purged, and that is somewhat contigent on getting the irqchip and timer stuff sorted (see my first attempt at that):
Quoted 4 lines
https://patchwork.kernel.org/patch/1852011/
Guess it depends where you want to draw the line on cleanups before something can be accepted..
I guess as long as we agree on where we are headed with the address translation, it's ok to just move the code now and change the code later. I would like to be strict about the include path stuff though.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Arnd Bergmann,
Quoted 13 lines · Arnd Bergmann
Quoted 13 lines
Also, FWIW, recall this related discussion and other possible DT binding:
http://www.spinics.net/lists/arm-kernel/msg219992.html
drivers/bus/orion-mbus.cfeels like the right option, but when I looked at it, getting the DT binding, and full dynamicness setup seemed like it would be best done after non-DT support was purged, and that is somewhat contigent on getting the irqchip and timer stuff sorted (see my first attempt at that):
Quoted 7 lines
Quoted 2 lines
Guess it depends where you want to draw the line on cleanups before something can be accepted..
I guess as long as we agree on where we are headed with the address translation, it's ok to just move the code now and change the code later. I would like to be strict about the include path stuff though.
Ok, so I could move the existing address decoding code into say drivers/bus/orion-mbus.c. Would that be ok for you?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 7 lines · Thomas Petazzoni
Quoted 4 lines
I guess as long as we agree on where we are headed with the address translation, it's ok to just move the code now and change the code later. I would like to be strict about the include path stuff though.
Ok, so I could move the existing address decoding code into say
drivers/bus/orion-mbus.c. Would that be ok for you?
Sure, sounds good to me.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Arnd BergmannDear Arnd Bergmann,
Quoted 10 lines · Arnd Bergmann
Quoted 5 lines
<
plat/pcie.h> is needed for a few PCIe functions shared with earlier families of Marvell SoC. My plan is that once this PCI driver gets accepted, I work on migrating the earlier Marvell SoC families to using this PCI driver, and therefore those functions would ultimately move in the driver indrivers/pci/host/, which would remove the <plat/pcie.h>.Hmm, although it is a bit unusual, I would actually propose duplicating that code for now, and merging a copy into the new driver right away. This gets rid of the header file dependency and lets you just delete the old file when the other platforms are converted.
Hum, why not, but I would definitely prefer to wait for the conversion of older platforms instead of duplicating this code. But if you feel like it's the right solution, I'll do it.
Quoted 9 lines
Quoted 5 lines
The <
mach/addr-map.h> is here to access the address decoding windows allocation/free API. And for this, there is no other long term plan than having an API provided by the platform code inarch/arm/, and used by drivers. Some other drivers may have to use this API as well in the future.This is harder to do, but I'm sure we can find a solution. At least the
addr-map.ccode has no other dependencies than theplat/addr-map.hheader, so we are fairly free to move it elsehwere.
The arch/arm/mach-mvebu/addr-map.c depends on arch/arm/plat-orion/addr-map.c, so any change on this will affect mach-kirkwood, mach-orion5x, mach-dove and mach-mv78xx0. As you can see, we have to take into account the existing code, and I don't think it's realistic to have a perfect solution immediately.
This address decoding code will continue to change for two reasons:
*) We are going to work on NOR/NAND support for mach-mvebu, and that
will also involve more interaction with the address decoding code.
*) As we are moving the earlier Marvell SoC families (mach-kirkwood,
mach-orion5x, mach-dove, mach-mv78xx0) to the Device Tree, this address decoding code will also evolve.
mach-mvebu is not a standalone new platform like mach-highbank for example, it relies on a lot of existing code from ealier platforms. It is nice to share existing code, but it also means that cleanups and refactoring take a lot more time.
I think we need to leave time for all these platforms to gradually converge and cleanup their infrastructure. It's not going to happen overnight.
Quoted 13 lines
I can think of several approaches that I'd prefer over your approach, although I have to admit that none of them makes me really happy:
a)
arch/arm/common/mv-addr-map.candarch/arm/include/asm/mach/mv-addr-map.hI assume that Russell will object to this one, but I let him speak for himselfb) drivers/bus/<something>
This would make a lot more sense if we followed the scheme I explained in my discussion to Jason Gunthorpe, where we basically treat this bus as a parent node in the device tree for anything that can get remapped. Without that change, it feels a little misplaced
Quoted 3 lines
c)
drivers/misc/
I usually object to anything put in here and would certainly prefer the previous one over this.
All of those approaches require a huge amount of work to convert the existing SoC families. Certainly, it will be done as time goes, and as older SoC families are converted to the DT and gradually converge inside mach-mvebu/, but if we have to wait for all of this to happen to get the PCIe support merged, it's not going to happen before several months (the PCIe stuff was originally posted on December, 7th, initially with the hope of targeting 3.9, the review and rework has taken a long time, so I'm now targeting 3.10 for the PCIe stuff, but I'd prefer not to have to postpone this to 3.11 and even 3.12 due to the heavy dependencies on address decoding rework).
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I don't understand this code with the masks and shifts. Could you add a comment here for readers like me?
Sure, will do.
Quoted 7 lines
Quoted 5 lines
It basically comes from the PCI-to-PCI bridge specification, which explains how the I/O address and I/O limit is split into two 16 bits registers, with those bizarre shifts and hardcoded values. I'll put a reference to the relevant section of the PCI-to-PCI bridge specification here.
Ok, I see. Thanks for the explanation.
I'll add a comment anyway, because it's true that it's a bit of magic going on here.
Quoted 24 lines
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +2
-0+ /* Get the I/O and memory ranges from DT */ + while ((range = of_pci_process_ranges(np, &res, range)) !=NULL) {
drivers/pci/host/pci-mvebu.c new file +2
-0+if (resource_type (&res) ==IORESOURCE_IO ) { +memcpy (&pcie->io, &res,sizeof (res));Quoted 8 lines
drivers/pci/host/pci-mvebu.c line 571
Quoted 5 lines
drivers/pci/host/pci-mvebu.c new file +5
-0+memcpy (&pcie->realio, &res,sizeof (res)); + pcie->io.name ="I/O" ; + pcie->realio.start &=0xFFFFF ; + pcie->realio.end &=0xFFFFF ; + }The bit masking seems fishy here. What exactly are you doing, does this just assume you have a 1MB window at most?
Basically, I have two resources for the I/O:
Quoted 7 lines
* One described in the DT, from
0xC0000000to0xC00FFFFFwhich will be used to create the address decoding windows for the I/O regions of the different PCIe interfaces. The PCI I/O virtual address0xffe00000will be mapped to those physical addresses. Those address decoding windows are configured with the special "remap" mechanism that ensures that if an access is made at0xC0000000+ offset, it will appear on the PCI bus as an I/O access at address "offset".
Quoted 4 lines
Right, I got this from reading the code. Unfortunately the
of_pci_process_rangesfunctions from your earlier patch makes this a little confusing as it marks this resource asIORESOURCE_IOwhen in reality it is theIORESOURCE_MEMresource that describes where in MMIO space the I/O window is.
Indeed. So maybe I should mark this resource as being IORESOURCE_MEM in the DT.
Quoted 16 lines
Quoted 11 lines
* One covering the low addresses
0x0->0xFFFFF(pcie->realio), which is used to tell the Linux PCI subsystem from which address range it should assign I/O addresses.Quoted 4 lines
Maybe something like
pcie->realio.start = 0; pcie->realio.end = pcie->io.end - pcie->io.start;Indeed, that would result in the same values. If you find it clearer, I'm fine with it.
Ideally we would read that from the ranges property as well, since it does not necessarily start at zero, although any other value would be a bit silly.
Quoted 7 lines
I definitely prefer the version I suggested over your version. On second thought I would make this
pcie->realio.start = PCIBIOS_MIN_IO; pcie->realio.end = min(pcie->io.end - pcie->io.start, IO_SPACE_LIMIT);to ensure that we are inside of the limits.
Ok, thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Thomas Petazzoni
Dear Arnd Bergmann,
Quoted 6 lines · Arnd Bergmann
Quoted 5 lines
<
plat/pcie.h> is needed for a few PCIe functions shared with earlier families of Marvell SoC. My plan is that once this PCI driver gets accepted, I work on migrating the earlier Marvell SoC families to using this PCI driver, and therefore those functions would ultimately move in the driver indrivers/pci/host/, which would remove the <plat/pcie.h>.
Quoted 8 lines
Quoted 4 lines
Hmm, although it is a bit unusual, I would actually propose duplicating that code for now, and merging a copy into the new driver right away. This gets rid of the header file dependency and lets you just delete the old file when the other platforms are converted.
Hum, why not, but I would definitely prefer to wait for the conversion of older platforms instead of duplicating this code. But if you feel like it's the right solution, I'll do it.
It's not something we do a lot, but in this case, I think it's better if it lets us avoid adding the platform specific include path, which I really want to avoid here.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
The <
mach/addr-map.h> is here to access the address decoding windows allocation/free API. And for this, there is no other long term plan than having an API provided by the platform code inarch/arm/, and used by drivers. Some other drivers may have to use this API as well in the future.This is harder to do, but I'm sure we can find a solution. At least the
addr-map.ccode has no other dependencies than theplat/addr-map.hheader, so we are fairly free to move it elsehwere.
Quoted 5 lines
The
arch/arm/mach-mvebu/addr-map.cdepends onarch/arm/plat-orion/addr-map.c, so any change on this will affect mach-kirkwood, mach-orion5x, mach-dove and mach-mv78xx0. As you can see, we have to take into account the existing code, and I don't think it's realistic to have a perfect solution immediately.
Yes, I realize this. I was thinking we would move all at least the file from plat-orion, and the header file. I don't care much whether we also move the platform specific setup from mach-*/addr-map.c, it works either way.
Quoted 9 lines
This address decoding code will continue to change for two reasons:
*) We are going to work on NOR/NAND support for mach-mvebu, and that
will also involve more interaction with the address decoding code.*) As we are moving the earlier Marvell SoC families (mach-kirkwood,
mach-orion5x, mach-dove, mach-mv78xx0) to the Device Tree, this address decoding code will also evolve.
Quoted 8 lines
mach-mvebu is not a standalone new platform like mach-highbank for example, it relies on a lot of existing code from ealier platforms. It is nice to share existing code, but it also means that cleanups and refactoring take a lot more time.
I think we need to leave time for all these platforms to gradually converge and cleanup their infrastructure. It's not going to happen overnight.
We don't need to do a complete overhaul of that code right now, but if we agree on a place where it can go, then I think we should move it now as just one extra patch to get rid of the header dependency. In the worst case, moving just the header file to include/linux would work, too.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
* One described in the DT, from
0xC0000000to0xC00FFFFFwhich will be used to create the address decoding windows for the I/O regions of the different PCIe interfaces. The PCI I/O virtual address0xffe00000will be mapped to those physical addresses. Those address decoding windows are configured with the special "remap" mechanism that ensures that if an access is made at0xC0000000+ offset, it will appear on the PCI bus as an I/O access at address "offset".
Quoted 7 lines
Quoted 4 lines
Right, I got this from reading the code. Unfortunately the
of_pci_process_rangesfunctions from your earlier patch makes this a little confusing as it marks this resource asIORESOURCE_IOwhen in reality it is theIORESOURCE_MEMresource that describes where in MMIO space the I/O window is.Indeed. So maybe I should mark this resource as being
IORESOURCE_MEMin the DT.
The DT seems fine here, just the code that interprets it is a little unusual. Maybe you can change the calling convention of that function to pass the type of resource you want as an argument?
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Arnd Bergmann,
Quoted 7 lines · Arnd Bergmann
Quoted 3 lines
Hum, why not, but I would definitely prefer to wait for the conversion of older platforms instead of duplicating this code. But if you feel like it's the right solution, I'll do it.
It's not something we do a lot, but in this case, I think it's better if it lets us avoid adding the platform specific include path, which I really want to avoid here.
Ok.
Quoted 10 lines
Quoted 5 lines
The
arch/arm/mach-mvebu/addr-map.cdepends onarch/arm/plat-orion/addr-map.c, so any change on this will affect mach-kirkwood, mach-orion5x, mach-dove and mach-mv78xx0. As you can see, we have to take into account the existing code, and I don't think it's realistic to have a perfect solution immediately.Yes, I realize this. I was thinking we would move all at least the file from plat-orion, and the header file. I don't care much whether we also move the platform specific setup from
mach-*/addr-map.c, it works either way.
Ok.
Quoted 5 lines
We don't need to do a complete overhaul of that code right now, but if we agree on a place where it can go, then I think we should move it now as just one extra patch to get rid of the header dependency. In the worst case, moving just the header file to
include/linuxwould work, too.
I'll try to cook something for this.
Quoted 6 lines
Quoted 2 lines
Indeed. So maybe I should mark this resource as being
IORESOURCE_MEMin the DT.The DT seems fine here, just the code that interprets it is a little unusual. Maybe you can change the calling convention of that function to pass the type of resource you want as an argument?
Erm? The type of the resource is encoded into the DT:
+ 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */
Quoted 1 line
From http://devicetree.org/Device_Tree_Usage#PCI_Host_Bridge:
phys.hi cell: npt000ss bbbbbbbb dddddfff rrrrrrrr
phys.mid cell: hhhhhhhh hhhhhhhh hhhhhhhh hhhhhhhh phys.low cell: llllllll llllllll llllllll llllllll
ss: space code
00: configuration space
01: I/O space
10: 32 bit memory space
11: 64 bit memory space
So the 0x81 at the beginning of the first line means I/O space, the 0x82 at the beginning of the second line means 32 bits memory space. The of_pci_process_ranges() function simply decodes those informations and fills the struct resource it returns with the appropriate resource type.
So passing the resource type as argument to of_pci_process_ranges() doesn't make much sense to me.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 25 lines · Thomas Petazzoni
Quoted 6 lines
Quoted 2 lines
Indeed. So maybe I should mark this resource as being
IORESOURCE_MEMin the DT.The DT seems fine here, just the code that interprets it is a little unusual. Maybe you can change the calling convention of that function to pass the type of resource you want as an argument?
Erm? The type of the resource is encoded into the DT:
(quoted diff) +2
-0+ 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */From http://devicetree.org/Device_Tree_Usage#PCI_Host_Bridge:
phys.hi cell: npt000ss bbbbbbbb dddddfff rrrrrrrr
phys.mid cell: hhhhhhhh hhhhhhhh hhhhhhhh hhhhhhhh phys.low cell: llllllll llllllll llllllll llllllllss: space code
00: configuration space
01: I/O space
10: 32 bit memory space
11: 64 bit memory space
Quoted 5 lines
So the
0x81at the beginning of the first line means I/O space, the0x82at the beginning of the second line means 32 bits memory space. Theof_pci_process_ranges()function simply decodes those informations and fills the struct resource it returns with the appropriate resource type.
Yes, of course. And the ranges property tells you how to turn the first address space into the second address space. So the above property defines that the PCI bus I/O space range from 0 to 0x10000 gets converted into the host MMIO range 0xc0000000 to 0xc0010000 on the host, and the PCI bus memory space range from 0 to 0x08000000 gets converted to the host MMIO range 0xc1000000 to 0xc9000000.
The output of your of_pci_process_ranges() function is the host MMIO range, not the range in the bus address space, so it has to be IORESOURCE_MEM.
Note that the memory range you have given in the example is a little unconventional. Usually we try to have an identity mapping between PCI bus memory space addresses and host MMIO space, which would result in a zero sys->mem_offset value in addition to the sys->io_offset value that is already zero.
Is there a limitation with the way that your address translation windows are set up? What I would hope you could do is a memory window like
<0x82000000 0 0xc1000000 0xc1000000 0 0x08000000>; /* non-prefetchable memory */
that translates the host MMIO range 0xc1000000-0xc9000000 into bus address 0xc1000000-0xc9000000 instead of bus address 0x00000000-0x08000000. Would it help if you switch the addresses around, so that the physical location of the memory space window is naturally aligned at 0xc0000000-0xc8000000 and the I/O space window is at 0xc8000000-0xc8100000?
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Arnd Bergmann,
Quoted 10 lines · Arnd Bergmann
Yes, of course. And the ranges property tells you how to turn the first address space into the second address space. So the above property defines that the PCI bus I/O space range from 0 to
0x10000gets converted into the host MMIO range0xc0000000to0xc0010000on the host, and the PCI bus memory space range from 0 to0x08000000gets converted to the host MMIO range0xc1000000to0xc9000000.The output of your
of_pci_process_ranges()function is the host MMIO range, not the range in the bus address space, so it has to beIORESOURCE_MEM.
I am sorry, but I don't get how this can work. My code currently relies on the DT encoding one I/O resource and one MEM resource to find which is one to use.
Right now, in the ranges property of my DT, I have:
0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */And the code parsing this does:
/* Get the I/O and memory ranges from DT */
while ((range = of_pci_process_ranges(np, &res, range)) != NULL) {
if (resource_type(&res) == IORESOURCE_IO) {
memcpy(&pcie->io, &res, sizeof(res));
memcpy(&pcie->realio, &res, sizeof(res));
pcie->io.name = "I/O";
pcie->realio.start &= 0xFFFFF;
pcie->realio.end &= 0xFFFFF;
}
if (resource_type(&res) == IORESOURCE_MEM) {
memcpy(&pcie->mem, &res, sizeof(res));
pcie->mem.name = "MEM";
}
}As you can see, it relies on one of the two ranges being for I/O (address starts with 0x81), and another range being for MEM (address starts with 0x82).
If I make both ranges start with 0x82, how can my code differentiate between the range used for I/O and the range used for MEM?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 15 lines · Thomas Petazzoni
Dear Arnd Bergmann,
Quoted 10 lines · Arnd Bergmann
Yes, of course. And the ranges property tells you how to turn the first address space into the second address space. So the above property defines that the PCI bus I/O space range from 0 to
0x10000gets converted into the host MMIO range0xc0000000to0xc0010000on the host, and the PCI bus memory space range from 0 to0x08000000gets converted to the host MMIO range0xc1000000to0xc9000000.The output of your
of_pci_process_ranges()function is the host MMIO range, not the range in the bus address space, so it has to beIORESOURCE_MEM.
Quoted 8 lines
I am sorry, but I don't get how this can work. My code currently relies on the DT encoding one I/O resource and one MEM resource to find which is one to use.
Right now, in the ranges property of my DT, I have:
0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */
This looks correct, yes, although as I suggested earlier it would be more conventional to use
0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0xc1000000 0xc1000000 0 0x08000000>; /* non-prefetchable memory */Quoted 17 lines
And the code parsing this does:
/* Get the I/O and memory ranges from DT */ while ((range = of_pci_process_ranges(np, &res, range)) != NULL) { if (resource_type(&res) == IORESOURCE_IO) { memcpy(&pcie->io, &res, sizeof(res)); memcpy(&pcie->realio, &res, sizeof(res)); pcie->io.name = "I/O"; pcie->realio.start &= 0xFFFFF; pcie->realio.end &= 0xFFFFF; } if (resource_type(&res) == IORESOURCE_MEM) { memcpy(&pcie->mem, &res, sizeof(res)); pcie->mem.name = "MEM"; } }
Quoted 6 lines
As you can see, it relies on one of the two ranges being for I/O (address starts with
0x81), and another range being for MEM (address starts with0x82).If I make both ranges start with
0x82, how can my code differentiate between the range used for I/O and the range used for MEM?
No, I mean the parser is wrong ;-)
I think the proper solution would be to change the parser to return both the host side resource, and another resource as the bus view. In the example above, the resources you get should be:
io_host = { .flags = IORESOURCE_MEM, .start = 0xc0000000, .end = 0xc000ffff };io_bus = { .flags = IORESOURCE_IO, .start = 0, .end = 0xffff };mem_host = { .flags = IORESOURCE_MEM, .start = 0xc1000000, .end = 0xc9ffffff };mem_bus = { .flags = IORESOURCE_MEM, .start = 0, .end = 0x7ffffff };
indicating that the PCI I/O space range 0-0xffff is mapped to the host physical address space at 0xc0000000-0xc000ffff, and that the PCI memory space range 0-0x7ffffff is mapped to host physical address space 0xc1000000-0xc9ffffff.
You really need all four resources to set up the mapping. For I/O space, you should do
pci_ioremap_io(io_bus->start, io_host->start);
and for the memory space, you need both to set up the correct mem_offset:
pci_add_resource_offset(&sys->resources, mem_host, mem_host->start - mem_bus->start);
If you do as I suggested above and set the memory window to an identity map at bus address 0xc1000000, the second argument to pci_add_resource_offset becomes zero, indicating that any access to the window results in a bus access at the same address.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 1 line · Thomas Petazzoni
+++
b/Documentation/devicetree/bindings/pci/mvebu-pci.txt
[..]
Quoted 24 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +24
-0+pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */ + 0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */ + 0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */ + 0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>;
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 64
Quoted 10 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +10
-0+ interrupt-map = <0x0800 0 0 1 &mpic 58 + 0x1000 0 0 1 &mpic 59 + 0x1800 0 0 1 &mpic 60 + 0x2000 0 0 1 &mpic 61 + 0x2800 0 0 1 &mpic 62 + 0x3000 0 0 1 &mpic 63 + 0x3800 0 0 1 &mpic 64 + 0x4000 0 0 1 &mpic 65 + 0x4800 0 0 1 &mpic 99 + 0x5000 0 0 1 &mpic 103>;
This is out of date now?
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 68
Quoted 3 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +3
-0+ pcie@0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>;
It would be great to get this sorted as per my prior comments.. Maybe like this is easy?
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";
// Index by marvell,pcie-port ?
regs = <0xd0040000 0x00002000
0xd0080000 0x00002000>;
ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ pcie@0,0 {
device_type = "pci";
reg = <0x0800 0 0 0>; // 00:01.0 (????)
marvell,pcie-port = <0>; };
}
It is abusive to map the device internal per-port registers through '0x00000800 0 0xd0040000' and 'reg' - that is not really the intent of the OF spec.
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Jason Gunthorpe,
Quoted 1 line · Jason Gunthorpe
This is out of date now?
Yes, will fix.
Quoted 21 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 68
Quoted 3 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>;It would be great to get this sorted as per my prior comments.. Maybe like this is easy?
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";// Index by marvell,pcie-port ?
regs = <0xd00400000x000020000xd0080000 0x00002000>;ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;
Quoted 6 lines
};
}It is abusive to map the device internal per-port registers through '
0x0000080000xd0040000' and 'reg' - that is not really the intent of the OF spec.
I am not sure to understand how this would work. Given a pcie at X,Y node, how would I find the address of the internal registers (i.e the ones at 0xd0040000, 0xd0080000) ?
You seem to propose a global regs = <...> property under pcie-controller, but indexing using marvell,pcie-port cannot work. PCIe interfaces are identified by two values (port,lane), so we have 0.0, 0.1, 0.2, 0.3, 1.0, 1.1, 1.2, 1.3, 2.0 and 3.0 on MV78460. I really would like to avoid having bizarre computations to find which entry in this big regs = <...> array correspond to a given PCIe interface.
Could you give a more detailed example, matching the PCIe DT data of the MV78460, which has many PCIe interfaces ?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 32 lines · Thomas Petazzoni
Quoted 21 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 68
Quoted 3 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>;It would be great to get this sorted as per my prior comments.. Maybe like this is easy?
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";// Index by marvell,pcie-port ?
regs = <0xd00400000x000020000xd0080000 0x00002000>;ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;Quoted 6 lines
};
}It is abusive to map the device internal per-port registers through '
0x0000080000xd0040000' and 'reg' - that is not really the intent of the OF spec.I am not sure to understand how this would work. Given a pcie at X,Y node, how would I find the address of the internal registers (i.e the ones at
0xd0040000,0xd0080000) ?
Quoted 6 lines
You seem to propose a global regs = <...> property under pcie-controller, but indexing using marvell,pcie-port cannot work. PCIe interfaces are identified by two values (port,lane), so we have 0.0, 0.1, 0.2, 0.3, 1.0, 1.1, 1.2, 1.3, 2.0 and 3.0 on MV78460. I really would like to avoid having bizarre computations to find which entry in this big regs = <...> array correspond to a given PCIe interface.
Well, using the 'global' regs is an easy way to avoid the complexity with address translation and 'assigned-addresses'. So some way to translate between link # and regs index is necessary. How about using the PCI device number? Device number 1 is regs entry 0, 2 is regs entry 1, etc.
The code flow would be then to iterate over DT children, extract the bus/dev/function from 'reg', ensure bus ==0, and index the global regs by device-1.
The other option is to try to make assigned-addresses work, Thierry had some comments on that, it is a bit convoluted too, IMHO.
Also note setting the device_type = "pci";
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 11 lines · Jason Gunthorpe
Quoted 6 lines
You seem to propose a global regs = <...> property under pcie-controller, but indexing using marvell,pcie-port cannot work. PCIe interfaces are identified by two values (port,lane), so we have 0.0, 0.1, 0.2, 0.3, 1.0, 1.1, 1.2, 1.3, 2.0 and 3.0 on MV78460. I really would like to avoid having bizarre computations to find which entry in this big regs = <...> array correspond to a given PCIe interface.
Well, using the 'global' regs is an easy way to avoid the complexity with address translation and 'assigned-addresses'. So some way to translate between link # and regs index is necessary. How about using the PCI device number? Device number 1 is regs entry 0, 2 is regs entry 1, etc.
Or you could use the "reg-names" property.
Arnd
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeDear Jason Gunthorpe,
Quoted 21 lines · Jason Gunthorpe
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 68
Quoted 3 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>;It would be great to get this sorted as per my prior comments.. Maybe like this is easy?
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";// Index by marvell,pcie-port ?
regs = <0xd00400000x000020000xd0080000 0x00002000>;ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;
Quoted 6 lines
};
}It is abusive to map the device internal per-port registers through '
0x0000080000xd0040000' and 'reg' - that is not really the intent of the OF spec.
The Device Tree would really look odd. We have one register range for each PCIe interface, but instead of nicely putting them inside the pcie at X,Y subnodes, we have a global regs = <..> property at the pcie-controller level? I can do that if you want, but it really sounds like the standard PCI DT bindings are horrible. Those register ranges are *per* PCIe interface, so any logical person would expect them inside the pcie at X,Y node...
But ok, if that's the way things should be, so be it.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 32 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
Quoted 21 lines · Jason Gunthorpe
Documentation/devicetree/bindings/pci/mvebu-pci.txt line 68
Quoted 3 lines
Documentation/devicetree/bindings/pci/mvebu-pci.txt new file +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>;It would be great to get this sorted as per my prior comments.. Maybe like this is easy?
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";// Index by marvell,pcie-port ?
regs = <0xd00400000x000020000xd0080000 0x00002000>;ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;Quoted 6 lines
};
}It is abusive to map the device internal per-port registers through '
0x0000080000xd0040000' and 'reg' - that is not really the intent of the OF spec.
Quoted 7 lines
The Device Tree would really look odd. We have one register range for each PCIe interface, but instead of nicely putting them inside the pcie at X,Y subnodes, we have a global regs = <..> property at the pcie-controller level? I can do that if you want, but it really sounds like the standard PCI DT bindings are horrible. Those register ranges are *per* PCIe interface, so any logical person would expect them inside the pcie at X,Y node...
I agree here. It seems that back at the time when the PCI DT bindings were devised no hardware existed with these properties. So I think that maybe it needs to be revised to take into account some of these new requirements. Just adding a bunch of properties to work around shortcomings in the specification doesn't seem very desirable to me.
That said, on Tegra there's an additional argument in favour of this because the addresses mapped through the ranges/reg properties are actually used to access the device's configuration space, which the 'ss' field in the first cell specifies. Quoting the DTS:
pcie-controller {
...
ranges = <0x00000800 0 0x80000000 0x80000000 0 0x00001000 /* port 0 registers */
0x00001000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */
0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */
0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
...
pci at 1,0 {
...
reg = <0x000800 0 0x80000000 0 0x1000>;
...
};
pci at 2,0 {
...
reg = <0x001000 0 0x80001000 0 0x1000>;
...
};
};So we're actually mapping the configuration space of pci at 1,0 and pci at 2,0 to the 0x80000000 and 0x80001000 addresses of the CPU physical address space. According to the PCI DT specification this isn't allowed, but it works on Linux and quite frankly I don't see why it shouldn't be allowed.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130306/543b0f8a/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 5 lines · Thierry Reding
I agree here. It seems that back at the time when the PCI DT bindings were devised no hardware existed with these properties. So I think that maybe it needs to be revised to take into account some of these new requirements. Just adding a bunch of properties to work around shortcomings in the specification doesn't seem very desirable to me.
Well, all the PCI hosts should use the same mechanism, and it should make sense in the context of the rest of the spec. So I think some approval/comment on this specific issue by Grant/Rob/Arnd is important at this point.
The way the spec is written, 'reg' in a 'device_type = pci' context is the device's config address - it is like 'reg' in a CPU block, or I2C bus - the value has nothing to do with CPU mappable memory.
Quoted 3 lines
space. According to the PCI DT specification this isn't allowed, but it works on Linux and quite frankly I don't see why it shouldn't be allowed.
AFAIK, this only works because you are not using 'device_type = pci'.
IMHO, 'device_type = pci' is not optional, so you need to come up with something that works under those conditions, and it changes how the OF core handles 'reg' and the like.
As I said before, the two best options to me seem to be:
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";
// Index by marvell,pcie-port/reg-names/etc
regs = <0xd0040000 0x00002000
0xd0080000 0x00002000>;
ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ pcie at 0,0 {
device_type = "pci";
reg = <0x0800 0 0 0>; // 00:01.0 (????)
marvell,pcie-port = <0>; };
}
or:
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";
ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000 /* non-prefetchable memory */
// Mapping for per-port host controller registers
0x82000000 0x10000000 0xd0040000 0xd0040000 0 0x2000>; pcie at 0,0 {
device_type = "pci";
reg = <0x0800 0 0 0>; // 00:01.0 (????)
marvell,pcie-port = <0>; assigned-addresses = <0 0 0 0 0 // BAR 0 of the bridge, unused
0 0 0 0 0 // BAR 1 of the bridge, unused 0x82000000 0x10000000 0xd0040000 0 0x2000>; // Extended
};
}
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 21 lines · Jason Gunthorpe
Quoted 5 lines · Thierry Reding
I agree here. It seems that back at the time when the PCI DT bindings were devised no hardware existed with these properties. So I think that maybe it needs to be revised to take into account some of these new requirements. Just adding a bunch of properties to work around shortcomings in the specification doesn't seem very desirable to me.
Well, all the PCI hosts should use the same mechanism, and it should make sense in the context of the rest of the spec. So I think some approval/comment on this specific issue by Grant/Rob/Arnd is important at this point.
The way the spec is written, 'reg' in a '
device_type= pci' context is the device's config address - it is like 'reg' in a CPU block, or I2C bus - the value has nothing to do with CPU mappable memory.Quoted 3 lines
space. According to the PCI DT specification this isn't allowed, but it works on Linux and quite frankly I don't see why it shouldn't be allowed.
Quoted 5 lines
AFAIK, this only works because you are not using '
device_type= pci'.IMHO, '
device_type= pci' is not optional, so you need to come up with something that works under those conditions, and it changes how the OF core handles 'reg' and the like.
Actually I do use device_type = "pciex" for the pci at 1,0 and pci at 2,0 nodes which was suggested back when I first posted this series. Yet the address translation still works properly. And it should since they are translated to the parent bus which is used to access the configuration space window for each root port.
Looking at the code it looks indeed like the OF core only matches on "pci" (and "vci"), but not "pciex". Perhaps I should change both nodes to device_type = "pci" and see what the results are.
Quoted 38 lines
As I said before, the two best options to me seem to be:
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";// Index by marvell,pcie-port/reg-names/etc
regs = <0xd00400000x000020000xd0080000 0x00002000>;ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;};
}or:
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";ranges = <0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000 /* non-prefetchable memory */ // Mapping for per-port host controller registers 0x82000000 0x10000000 0xd0040000 0xd0040000 0 0x2000>;pcie at 0,0 { device_type = "pci"; reg = <0x0800 0 0 0>; // 00:01.0 (????) marvell,pcie-port = <0>;assigned-addresses = <0 0 0 0 0 // BAR 0 of the bridge, unused
0 0 0 0 0 // BAR 1 of the bridge, unused0x820000000x100000000xd004000000x2000>; // Extended
Quoted 6 lines
};
}Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
However that would probably not work out-of-the-box with the current OF core because of_address_to_resource() won't know how to find the new property. The assigned-addresses alternative seems to be the only one that would work on the current OF core and achieve proper address translation. But it doesn't seem like a good solution either since it repurposes the meaning of the property and therefore isn't any better than encoding the same information in the reg property.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130307/989c145c/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 5 lines · Thierry Reding
Actually I do use
device_type= "pciex" for the pci at 1,0 and pci at 2,0 nodes which was suggested back when I first posted this series. Yet the address translation still works properly. And it should since they are translated to the parent bus which is used to access the configuration space window for each root port.
Right - pciex != pci - as you note:
Quoted 3 lines
Looking at the code it looks indeed like the OF core only matches on "pci" (and "vci"), but not "pciex". Perhaps I should change both nodes to
device_type= "pci" and see what the results are.
The string 'pci' engages the special behavior, pciex isn't a valid keyword.
This is why I keep saying these bindings need to use 'device_type = "pci"' :)
Quoted 9 lines
Quoted 3 lines
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
You are thinking a new property 'host-controller-regs' or the like?
Quoted 7 lines
However that would probably not work out-of-the-box with the current OF core because
of_address_to_resource()won't know how to find the new property. The assigned-addresses alternative seems to be the only one that would work on the current OF core and achieve proper address translation. But it doesn't seem like a good solution either since it repurposes the meaning of the property and therefore isn't any better than encoding the same information in the reg property.
Except that reg is specifically not supposed to be handling CPU bus addresses and assigned-addresses is.
Adding a hidden non-standard BAR to model the hidden non-standard memory region associated with the bridge is a stretch, but it is not a very big stretch...
In any event, regs should not be used, and something needs to be decided!
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Jason Gunthorpe
On Thu, Mar 07, 2013 at 09:08:32AM +0100, Thierry Reding wrote:
[...]
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
You are thinking a new property 'host-controller-regs' or the like?
Well, something shorter would be nice, but that's the general idea, yes. As I mentioned before, for Tegra these registers aren't actually any controller specific registers but rather a window to access the PCI configuration space for the root ports.
Going via the FPCI mapping of the configuration space doesn't generate transactions on bus 0, which is why there are extra windows for each root port. That's why I was arguing about reusing the reg property in the first place because it actually represents the configuration space.
But this is likely to be unique to Tegra and Marvell hardware doesn't behave this way, but instead needs the registers for a different purpose. Something like host-controller-regs might be more appropriate.
Quoted 11 lines
Quoted 7 lines
However that would probably not work out-of-the-box with the current OF core because
of_address_to_resource()won't know how to find the new property. The assigned-addresses alternative seems to be the only one that would work on the current OF core and achieve proper address translation. But it doesn't seem like a good solution either since it repurposes the meaning of the property and therefore isn't any better than encoding the same information in the reg property.Except that reg is specifically not supposed to be handling CPU bus addresses and assigned-addresses is.
Quoted 6 lines
Adding a hidden non-standard BAR to model the hidden non-standard memory region associated with the bridge is a stretch, but it is not a very big stretch...
In any event, regs should not be used, and something needs to be decided!
I don't think assigned-addresses is a good fit either. The PCI binding document is equally specific about it as it is about the reg property. So in my opinion a separate property would be a better choice. The only big obstacle is that it needs to be somehow hooked up with the OF core so that proper address translation can be performed.
One possible solution that wouldn't be too hard to implement is to provide a new function (say of_get_named_address()) similar to of_get_address() which doesn't get the name of the register property from the struct of_bus but from a parameter and call that function from another new function similar to of_address_to_resource() that also gets the property name from a parameter. I can't think of a better name for the latter than of_named_address_to_resource(), which is rather long.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130307/44e758d7/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 13 lines · Thierry Reding
Quoted 1 line · Jason Gunthorpe
On Thu, Mar 07, 2013 at 09:08:32AM +0100, Thierry Reding wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
Quoted 6 lines
Quoted 1 line
You are thinking a new property 'host-controller-regs' or the like?
Well, something shorter would be nice, but that's the general idea, yes. As I mentioned before, for Tegra these registers aren't actually any controller specific registers but rather a window to access the PCI configuration space for the root ports.
Yes, I understand - but in this DT model configuration space access is a host controller function, not a PCI-device function. Anyhow I was also thinking that by the choice of the name it could do translation from the host-controller scope, not from the bridge scope - so the extra elements in ranges could be avoided as well. Hence the name..
Quoted 5 lines
I don't think assigned-addresses is a good fit either. The PCI binding document is equally specific about it as it is about the reg property. So in my opinion a separate property would be a better choice. The only big obstacle is that it needs to be somehow hooked up with the OF core so that proper address translation can be performed.
Yes, agreed.
My suggestion is to get the OF experts like GKL/Rob H/etc to weigh in on a preferred approach to this problem with the goal of standardizing across all PCI host drivers. Seems like there are 2 main options (outside regs + regnames/etc or new 'regs' in the bridge) and 1 hacky one (assigned addresses)
Quoted 7 lines
One possible solution that wouldn't be too hard to implement is to provide a new function (say
of_get_named_address()) similar toof_get_address()which doesn't get the name of the register property from the structof_busbut from a parameter and call that function from another new function similar toof_address_to_resource()that also gets the property name from a parameter. I can't think of a better name for the latter thanof_named_address_to_resource(), which is rather long.
Seems like a reasonable API, maybe pass in a be32*/length pointer instead of a name to be more flexible?
Cheers,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 33 lines · Jason Gunthorpe
Quoted 13 lines · Thierry Reding
Quoted 1 line · Jason Gunthorpe
On Thu, Mar 07, 2013 at 09:08:32AM +0100, Thierry Reding wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
Quoted 6 lines
Quoted 1 line
You are thinking a new property 'host-controller-regs' or the like?
Well, something shorter would be nice, but that's the general idea, yes. As I mentioned before, for Tegra these registers aren't actually any controller specific registers but rather a window to access the PCI configuration space for the root ports.
Yes, I understand - but in this DT model configuration space access is a host controller function, not a PCI-device function. Anyhow I was also thinking that by the choice of the name it could do translation from the host-controller scope, not from the bridge scope - so the extra elements in ranges could be avoided as well. Hence the name..
Quoted 5 lines
I don't think assigned-addresses is a good fit either. The PCI binding document is equally specific about it as it is about the reg property. So in my opinion a separate property would be a better choice. The only big obstacle is that it needs to be somehow hooked up with the OF core so that proper address translation can be performed.
Quoted 7 lines
Yes, agreed.
My suggestion is to get the OF experts like GKL/Rob H/etc to weigh in on a preferred approach to this problem with the goal of standardizing across all PCI host drivers. Seems like there are 2 main options (outside regs + regnames/etc or new 'regs' in the bridge) and 1 hacky one (assigned addresses)
Arnd is already Cc'ed on this thread, adding Grant, Rob and the devicetree-discuss ML.
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
1) Use a "regs" property outside of the root port nodes with
some mechanism to index into them from within the root port
nodes. Conceptually somewhat like this:
pcie-controller {
...
regs = <0x80000000 0x00001000
0x80001000 0x00001000>;
pci at 0,1 {
...
port-index = <0>;
};
pci at 0,2 {
...
port-index = <1>;
};
};
2) Use a "regs" property inside of the root part nodes, along
the following lines:
pcie-controller {
...
pci at 0,1 {
...
reg = <0x00000800 0 0 0 0>;
regs = <0x80000000 0x00001000>;
};
pci at 0,2 {
...
reg = <0x00001000 0 0 0 0>;
regs = <0x80001000 0x00001000>;
};
};
3) Repurpose the "assigned-addresses" property to achieve the
same. This should work out-of-the-box but isn't a good fit
because it conflicts with the OF PCI specification which
defines this property to contain the addresses assigned to
the base address registers.Options 1 and 2 above require changes to the OF core to allow proper address translation, but the changes shouldn't be very big.
Quoted 10 lines
Quoted 7 lines
One possible solution that wouldn't be too hard to implement is to provide a new function (say
of_get_named_address()) similar toof_get_address()which doesn't get the name of the register property from the structof_busbut from a parameter and call that function from another new function similar toof_address_to_resource()that also gets the property name from a parameter. I can't think of a better name for the latter thanof_named_address_to_resource(), which is rather long.Seems like a reasonable API, maybe pass in a be32*/length pointer instead of a name to be more flexible?
There's already __of_address_to_resource() which takes a be32 * and size but I thought it might be easier to wrap that to make it easier on the drivers to use the API.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130307/2df62bd0/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 42 lines · Thierry Reding
Quoted 33 lines · Jason Gunthorpe
Quoted 13 lines · Thierry Reding
Quoted 1 line · Jason Gunthorpe
On Thu, Mar 07, 2013 at 09:08:32AM +0100, Thierry Reding wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Both have various problems, but I think I prefer the first one as it doesn't conflate the contoller registers and host apertures in a single ranges..
I think a better alternative would be (and this matches what Thomas has said elsewhere) to use something like the first alternative but move the regs property into the pcie at 0,X nodes. That would save us from having to index a property in the parent. At least from a DT point of view I find that to be a more consistent representation.
Quoted 6 lines
Quoted 1 line
You are thinking a new property 'host-controller-regs' or the like?
Well, something shorter would be nice, but that's the general idea, yes. As I mentioned before, for Tegra these registers aren't actually any controller specific registers but rather a window to access the PCI configuration space for the root ports.
Yes, I understand - but in this DT model configuration space access is a host controller function, not a PCI-device function. Anyhow I was also thinking that by the choice of the name it could do translation from the host-controller scope, not from the bridge scope - so the extra elements in ranges could be avoided as well. Hence the name..
Quoted 5 lines
I don't think assigned-addresses is a good fit either. The PCI binding document is equally specific about it as it is about the reg property. So in my opinion a separate property would be a better choice. The only big obstacle is that it needs to be somehow hooked up with the OF core so that proper address translation can be performed.
Quoted 7 lines
Yes, agreed.
My suggestion is to get the OF experts like GKL/Rob H/etc to weigh in on a preferred approach to this problem with the goal of standardizing across all PCI host drivers. Seems like there are 2 main options (outside regs + regnames/etc or new 'regs' in the bridge) and 1 hacky one (assigned addresses)
Quoted 8 lines
Arnd is already Cc'ed on this thread, adding Grant, Rob and the devicetree-discuss ML.
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Rob
Quoted 67 lines
1) Use a "regs" property outside of the root port nodes with some mechanism to index into them from within the root port nodes. Conceptually somewhat like this: pcie-controller { ... regs = <0x80000000 0x00001000 0x80001000 0x00001000>; pci at 0,1 { ... port-index = <0>; }; pci at 0,2 { ... port-index = <1>; }; }; 2) Use a "regs" property inside of the root part nodes, along the following lines: pcie-controller { ... pci at 0,1 { ... reg = <0x00000800 0 0 0 0>; regs = <0x80000000 0x00001000>; }; pci at 0,2 { ... reg = <0x00001000 0 0 0 0>; regs = <0x80001000 0x00001000>; }; }; 3) Repurpose the "assigned-addresses" property to achieve the same. This should work out-of-the-box but isn't a good fit because it conflicts with the OF PCI specification which defines this property to contain the addresses assigned to the base address registers.Options 1 and 2 above require changes to the OF core to allow proper address translation, but the changes shouldn't be very big.
Quoted 10 lines
Quoted 7 lines
One possible solution that wouldn't be too hard to implement is to provide a new function (say
of_get_named_address()) similar toof_get_address()which doesn't get the name of the register property from the structof_busbut from a parameter and call that function from another new function similar toof_address_to_resource()that also gets the property name from a parameter. I can't think of a better name for the latter thanof_named_address_to_resource(), which is rather long.Seems like a reasonable API, maybe pass in a be32*/length pointer instead of a name to be more flexible?
There's already
__of_address_to_resource()which takes a be32 * and size but I thought it might be easier to wrap that to make it easier on the drivers to use the API.Thierry
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 8 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
Adding a property in the root port nodes seems like a cleaner and more accurate representation of the hardware to me, but if that's not acceptable perhaps we need to bite the bullet and add the code to look the registers up from the parent's reg property.
Thierry
Quoted 40 lines
Quoted 40 lines
1) Use a "regs" property outside of the root port nodes with some mechanism to index into them from within the root port nodes. Conceptually somewhat like this: pcie-controller { ... regs = <0x80000000 0x00001000 0x80001000 0x00001000>; pci at 0,1 { ... port-index = <0>; }; pci at 0,2 { ... port-index = <1>; }; }; 2) Use a "regs" property inside of the root part nodes, along the following lines: pcie-controller { ... pci at 0,1 { ... reg = <0x00000800 0 0 0 0>; regs = <0x80000000 0x00001000>; }; pci at 0,2 { ... reg = <0x00001000 0 0 0 0>; regs = <0x80001000 0x00001000>; }; };
Quoted 8 lines
Quoted 8 lines
3) Repurpose the "assigned-addresses" property to achieve the same. This should work out-of-the-box but isn't a good fit because it conflicts with the OF PCI specification which defines this property to contain the addresses assigned to the base address registers.
Options 1 and 2 above require changes to the OF core to allow proper address translation, but the changes shouldn't be very big.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130308/17196dd1/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 6 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
Quoted 8 lines
Quoted 2 lines
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
Thomas just posted an implementation like this, please see:
http://www.spinics.net/lists/arm-kernel/msg228749.html
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 19 lines · Jason Gunthorpe
Quoted 9 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 6 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
Quoted 8 lines
Quoted 2 lines
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
Quoted 3 lines
Thomas just posted an implementation like this, please see:
Oh well. I don't like it, but if that's the way it has to be, then so be it. Any reason why the reg-names can't match the root port's node name so that it can be used directly instead of going through hoops to construct a string from extra parameters? Like below:
pcie-controller {
regs = <...>;
reg-names = ..., "pcie at 1,0", "pcie at 2,0";
pcie at 1,0 {
...
};
pcie at 2,0 {
...
};
};Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130308/c48fa649/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 20 lines · Thierry Reding
Quoted 19 lines · Jason Gunthorpe
Quoted 17 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 8 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
Quoted 3 lines
Quoted 3 lines
Thomas just posted an implementation like this, please see:
The example in that posting looks messed up to me.
1) It has "reg = <0x0800 0 0 0 0>", but 0x0800 0 0 is not a valid address in the address space defined by its parent - because the form of the parent's ranges property indicates that it's a PCI-style address form. 0x0800 0 0 lacks the top bits that indicate non-relocatable and the type (I/O, memory, etc).
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <0x0800 0 0 0 0> and <0x1000 0 0 0 0> using any rule that I know.
What is address space represented by the pcie-controller node? reg properties in child nodes must contain proper addresses in some well-defined address space. It looks like this example is trying to put half of the PCI bus abstraction in the pcie-controller node and the rest in the pcie sub-nodes, with some handwaving address-space splicing between the two. That's not kosher. Each level of the tree needs a coherent address space definition, otherwise the address space "calculus" breaks.
Quoted 28 lines
Oh well. I don't like it, but if that's the way it has to be, then so be it. Any reason why the reg-names can't match the root port's node name so that it can be used directly instead of going through hoops to construct a string from extra parameters? Like below:
pcie-controller { regs = <...>; reg-names = ..., "pcie at 1,0", "pcie at 2,0"; pcie at 1,0 { ... }; pcie at 2,0 { ... }; };Thierry
_______________________________________________
devicetree-discuss mailing list
devicetree-discuss at lists.ozlabs.org
https://lists.ozlabs.org/listinfo/devicetree-discuss
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Mitch Bradley
Quoted 1 line
The example in that posting looks messed up to me.
1) It has "reg = <
0x08000 0 0 0>", but0x08000 0 is not a valid address in the address space defined by its parent - because the form of the parent's ranges property indicates that it's a PCI-style address form.0x08000 0 lacks the top bits that indicate non-relocatable and the type (I/O, memory, etc).
You need to review the OF PCI bindings to make sense of this. The subnodes are PCI devices. Those PCI devices show up in the configuration space (ie lspci). They are the PCI root port bridges.
The reg value follows the OF PCI spec and has the configuration address of the bridge. For the first port's bridge this address in lspci format is 00:01.0 which encodes to <0x800 0 0>
The Linux OF PCI core uses the reg value in this format to match the OF node in the DT to the PCI device node as it does PCI discovery.
Quoted 2 lines
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <
0x08000 0 0 0> and <0x10000 0 0 0> using any rule that I know.
@0,1,0 (bus,device,fn) could be more appropriate, but that is cosmetic?
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Jason Gunthorpe
On Fri, Mar 08, 2013 at 09:43:11AM -1000, Mitch Bradley wrote:
[...]
Quoted 5 lines
Quoted 2 lines
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <
0x08000 0 0 0> and <0x10000 0 0 0> using any rule that I know.@0,1,0 (bus,device,fn) could be more appropriate, but that is cosmetic?
The OF PCI specification is pretty strict about this as well. It says in section 2.2.1.3. that only the DD and DD,FF forms can appear in a device path.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130308/82f997bf/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Thierry Reding,
Quoted 6 lines · Thierry Reding
Quoted 1 line · Jason Gunthorpe
On Fri, Mar 08, 2013 at 09:43:11AM -1000, Mitch Bradley wrote:
[...]
Quoted 3 lines
Quoted 2 lines
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <
0x08000 0 0 0> and <0x10000 0 0 0> using any rule that I know.
Quoted 6 lines
Quoted 2 lines
@0,1,0 (bus,device,fn) could be more appropriate, but that is cosmetic?
The OF PCI specification is pretty strict about this as well. It says in section 2.2.1.3. that only the DD and DD,FF forms can appear in a device path.
Note that in the case of my driver, the @X,Y represent the port and lane of the PCIe interface. Not sure if it is correct, I can change the names to whatever is appropriate, those names aren't used anywhere in the driver.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 12 lines · Thomas Petazzoni
Dear Thierry Reding,
Quoted 9 lines · Thierry Reding
Quoted 1 line · Jason Gunthorpe
On Fri, Mar 08, 2013 at 09:43:11AM -1000, Mitch Bradley wrote:
[...]
Quoted 5 lines
Quoted 2 lines
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <
0x08000 0 0 0> and <0x10000 0 0 0> using any rule that I know.@0,1,0 (bus,device,fn) could be more appropriate, but that is cosmetic?
Quoted 8 lines
Quoted 3 lines
The OF PCI specification is pretty strict about this as well. It says in section 2.2.1.3. that only the DD and DD,FF forms can appear in a device path.
Note that in the case of my driver, the @X,Y represent the port and lane of the PCIe interface. Not sure if it is correct, I can change the names to whatever is appropriate, those names aren't used anywhere in the driver.
I think they need to be changed to be @dev,fn in order to conform to the OF PCI binding.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130311/40ace6a4/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeQuoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Mitch Bradley
Quoted 1 line
The example in that posting looks messed up to me.
1) It has "reg = <
0x08000 0 0 0>", but0x08000 0 is not a valid address in the address space defined by its parent - because the form of the parent's ranges property indicates that it's a PCI-style address form.0x08000 0 lacks the top bits that indicate non-relocatable and the type (I/O, memory, etc).
Quoted 3 lines
You need to review the OF PCI bindings to make sense of this. The subnodes are PCI devices. Those PCI devices show up in the configuration space (ie lspci). They are the PCI root port bridges.
As it turns out, I wrote those bindings, almost 20 years ago.
Having dug through old versions of that patch, I think I see the source of the confusion.
In a very early version of the patch -
http://www.spinics.net/lists/arm-kernel/msg211839.html
the pcie-controller address space had 1 address cell, essentially propagating the CPU address down to the subordinate nodes, and the subordinate pcie nodes had no child address specification. Somebody correctly observed that PCI buses need to export 3/2 address/size cells to their children.
But that stipulation applies only to the subordinate pcie nodes, not necessarily to the enclosing pcie-controller node.
A later version -
http://permalink.gmane.org/gmane.linux.kernel.pci/20358
applied 3/2 PCI addressing at both levels. That is somewhat sensible if you treat the top level as a "PCI root complex" and if the subordinate port control registers are really PCI config header blocks.
But look at the ranges entries therein. PCI config address <0x800 ..> maps to CPU address 0xd004000 size 0x2000 and <0x1000 ..> maps to 0xd0044000 size 0x2000. 0x800 size 0x2000 encroaches into the <0x1000 ..> range. That can't be right.
That, plus the first version of the patch, makes me think that these root-port control registers might not really be PCI config registers, in which case the use of PCI addressing at the top level is a fiction.
Furthermore, we have the fact that pcie at 0,0 corresponds to marvell,pcie-port = <0> and marvell,pcie-lane = <0>, and similarly for pcie at 0,1 and pcie at 0,2. That correspondence still appears in the recent patch.
So it looks like the "@0,0" addressing attempts to represent lane,port instead of device,function PCI config space addressing. 0,0 is not the right PCI device/function number for reg=<0x800 ...> - 0x800 is device 1, not device 0.
Another problem is the device_type values. If the top level is to be interpreted as a PCI addressing domain, it should have a device_type=pci property.
One solution is for the top node to use 1-cell addresses, passing the CPU address to the subordinates. The subordinate nodes use the standard PCI address representation. The subordinate reg properties just list the CPU address of the control registers. Each subordinate has a ranges properties to translate PCI to CPU addresses and a bus-range property for PCI bus numbers (unless those are determined dynamically).
What you lose with that is the ability to refer to the nodes by port,lane. If that is important, then the top address domain needs an additional cell to say whether a given address is a CPU address or port,lane. The top node would need a ranges to map the various forms into CPU addresses. The child reg properties could use the port,lane form, and marvell,pcie-{port,lane} would disappear.
In neither case would you need the controversial "reg-names" thing.
The tradeoff:
Existing proposed patch: Violates addressing rules, requires funny new "reg-names" mechanism, pretends to create a PCI bus level that does not exist.
One-cell top address: Follows addressing rules, may be able to use existing "simple-bus" address translation code, does not permit the pretty @lane,port human-readable form.
Two-cell top address: Follows addressing rules, possibly requires new code to translate this particular 2-address-cell format, permits @lane,port representation.
Quoted 5 lines
The reg value follows the OF PCI spec and has the configuration address of the bridge. For the first port's bridge this address in lspci format is 00:01.0 which encodes to <
0x8000 0>
Quoted 8 lines
The Linux OF PCI core uses the reg value in this format to match the OF node in the DT to the PCI device node as it does PCI discovery.
Quoted 2 lines
2) The "@0,0" and "@1,0" suffixes do not correspond to the reg values <
0x08000 0 0 0> and <0x10000 0 0 0> using any rule that I know.@0,1,0 (bus,device,fn) could be more appropriate, but that is cosmetic?
Except that the @0,0 in this case seems to represent port,lane and not device,function , as argued above.
Quoted 3 lines
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 13 lines · Mitch Bradley
Quoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Mitch Bradley
Quoted 1 line
The example in that posting looks messed up to me.
1) It has "reg = <
0x08000 0 0 0>", but0x08000 0 is not a valid address in the address space defined by its parent - because the form of the parent's ranges property indicates that it's a PCI-style address form.0x08000 0 lacks the top bits that indicate non-relocatable and the type (I/O, memory, etc).
Quoted 8 lines
Quoted 3 lines
You need to review the OF PCI bindings to make sense of this. The subnodes are PCI devices. Those PCI devices show up in the configuration space (ie lspci). They are the PCI root port bridges.
As it turns out, I wrote those bindings, almost 20 years ago.
Having dug through old versions of that patch, I think I see the source of the confusion.
Not sure, I think this has gone into a weird tangent, your conclusions don't match how the PCI-E root complex is presented to the kernel.
Lets just go back to the start?
The pci-controller node is a root complex.
The children of that node are elements in the root complex - they are the root port bridges. They are all on PCI bus 0.
Each root port bridge responds to configuration cycles, and has a PCI configuration space. They show up in lspci.
The host controller device driver knows how to issue configuration cycles, and it knows how to deliver configuration cycles on bus 0 to the bridges.
This one:
+ pcie at 0,0 {
+ device_type = "pci";
+ reg = <0x0800 0 0 0 0>;
Responds to bus 0, device 1, function 0.
Each root port bridge has an associated non-PCI MMIO address space. The child above is associated with 0xd0040000. This MMIO address space is needed by the host controller driver to operate the port.
The text after the @ is a mistake, Linux doesn't enforce anything about this text so nobody noticed till now, it should be corrected to @1,0, similarly for the second one.
Also, from your comments the top level should gain a 'device_type = "pci"'.
So, from that point, does the DT start to make sense? Are there other problems?
The question at hand is how do you associate the non-PCI MMIO space 0xd0040000 with the root port bridge 'pcie at 1,0' ? The patch proposes to use a reg array in the controller plus reg-names to do this.
I have trouble making sense of your suggestions to switch away from 5dw addressing. It is critical that the DT child stanza be associated with the PCI discovered root port bridge. By my understanding Linux does this based on the configuration space format 'reg' value. How is that association possible if you switch things away from 5dw addressing?
Also, this is for firmware-less embedded. Linux is required to scan the bus and do full address assignment of all PCI devies, including the root port bridges. The ranges property on the top node specifies the addressing apertures that are available for this process. This is common practice, there are many examples of this in kernel dts for PCI controllers.
It is impossible to specify any detail for the bridges (be it address ranges or bus number ranges) because we have no idea what discovery will find, or what values Linux will choose to assign.
Quoted 1 line
In neither case would you need the controversial "reg-names" thing.
reg-names has been a standard feature in the Linux kernel for a bit already..
Thanks a lot for looking at this,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Jason Gunthorpe
Quoted 9 lines · Mitch Bradley
Quoted 8 lines · Jason Gunthorpe
Quoted 6 lines · Mitch Bradley
Quoted 1 line
The example in that posting looks messed up to me.
1) It has "reg = <
0x08000 0 0 0>", but0x08000 0 is not a valid address in the address space defined by its parent - because the
form of
Quoted 63 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
the parent's ranges property indicates that it's a PCI-style address form.
0x08000 0 lacks the top bits that indicate non-relocatable and the type (I/O, memory, etc).Quoted 8 lines
Quoted 3 lines
You need to review the OF PCI bindings to make sense of this. The subnodes are PCI devices. Those PCI devices show up in the configuration space (ie lspci). They are the PCI root port bridges.
As it turns out, I wrote those bindings, almost 20 years ago.
Having dug through old versions of that patch, I think I see the source of the confusion.
Not sure, I think this has gone into a weird tangent, your conclusions don't match how the PCI-E root complex is presented to the kernel.
Lets just go back to the start?
The pci-controller node is a root complex.
The children of that node are elements in the root complex - they are the root port bridges. They are all on PCI bus 0.
Each root port bridge responds to configuration cycles, and has a PCI configuration space. They show up in lspci.
The host controller device driver knows how to issue configuration cycles, and it knows how to deliver configuration cycles on bus 0 to the bridges.
This one:
(quoted diff) +3
-0+ pcie at 0,0 { + device_type = "pci"; + reg = <0x0800 0 0 0 0>;Responds to bus 0, device 1, function 0.
Each root port bridge has an associated non-PCI MMIO address space. The child above is associated with
0xd0040000. This MMIO address space is needed by the host controller driver to operate the port.The text after the @ is a mistake, Linux doesn't enforce anything about this text so nobody noticed till now, it should be corrected to @1,0, similarly for the second one.
Also, from your comments the top level should gain a '
device_type= "pci"'.So, from that point, does the DT start to make sense? Are there other problems?
The question at hand is how do you associate the non-PCI MMIO space
0xd0040000with the root port bridge 'pcie at 1,0' ? The patch proposes to use a reg array in the controller plus reg-names to do this.I have trouble making sense of your suggestions to switch away from 5dw addressing. It is critical that the DT child stanza be associated with the PCI discovered root port bridge. By my understanding Linux does this based on the configuration space format 'reg' value. How is that association possible if you switch things away from 5dw addressing?
Quoted 6 lines
Also, this is for firmware-less embedded. Linux is required to scan the bus and do full address assignment of all PCI devies, including the root port bridges. The ranges property on the top node specifies the addressing apertures that are available for this process. This is common practice, there are many examples of this in kernel dts for PCI controllers.
Okay, I think I finally get it. The Marvell root port bridge setup registers look like standard config headers, even though they aren't really in config space (because you need a different access method for them, compared to real config accesses to downstream devices).
So, in order for the common code that enumerates the PCI bus to work, you need to "spoof" the config accesses so that when you try to access those particular "pseudo config headers", it uses the MMIO method instead of the real config access method.
If that is indeed the case, them I would vote for a slight modification of the intermediate patch that I cited earlier - the one in which the ranges property includes translations from those special config addresses into CPU addresses. The modification is to fix the sizes, changing 0x2000 to 0x800, so those ranges entries do not overlap in the child address domain.
With that change, plus fixing the "@port,lane" strings to be "@dev,fun", the DT would not raise all the red flags that led me down this rathole.
The controller driver would still have to implement the magic spoofing of those funky config headers, but at least it would be doing it based on well-established representation principles, specifically ranges entries and well-formed child addresses, without having to resort to the new sideband "port and lane" address representation.
Here's what it would look like:
pcie-controller {
compatible = "marvell,armada-370-pcie";
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
bus-range = <0x00 0xff>;
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root
port config header */
0x00001000 0 0 0xd4008000 0 0x00000800 /* Root
port config header */
0x82000000 0 0xe0000000 0xe0000000 0 0x08000000 /*
non-prefetchable memory */
0x81000000 0 0 0xe8000000 0 0x00100000>; /*downstream I/O */
pcie at 1,0 {
device_type = "pci";
reg = <0x0800 0 0 0 0>;
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
interrupt-map = <0 0 0 0 &mpic 58>;
marvell,pcie-port = <0>;
marvell,pcie-lane = <0>;
clocks = <&gateclk 5>;
status = "disabled";
};
pcie at 2,0 {
device_type = "pci";
reg = <0x1000 0 0 0 0>;
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
interrupt-map = <0 0 0 0 &mpic 62>;
marvell,pcie-port = <1>;
marvell,pcie-lane = <0>;
clocks = <&gateclk 9>;
status = "disabled";
};
};Quoted 13 lines
It is impossible to specify any detail for the bridges (be it address ranges or bus number ranges) because we have no idea what discovery will find, or what values Linux will choose to assign.
Quoted 1 line
In neither case would you need the controversial "reg-names" thing.
reg-names has been a standard feature in the Linux kernel for a bit already..
Thanks a lot for looking at this,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Mitch Bradley
Okay, I think I finally get it. The Marvell root port bridge setup registers look like standard config headers, even though they aren't really in config space (because you need a different access method for them, compared to real config accesses to downstream devices).
Well, thats pretty close. On Marvell the MMIO space has a lot of stuff, some of it is config related, lots of it isn't. In fact most of it isn't. Tegra is different, the MMIO region is exactly the config space of the root port bridge.
Tegra could probably happily and sanely do as you've described (well, see below), but it is wonky for Marvell. It looks like there are more drivers coming that have this need - so I had hoped for a general answer to the 'per-port MMIO space' problem that is unrelated to the MMIO spaces role in config access.
Quoted 4 lines
So, in order for the common code that enumerates the PCI bus to work, you need to "spoof" the config accesses so that when you try to access those particular "pseudo config headers", it uses the MMIO method instead of the real config access method.
There are rules for config access in both Tegra and Marvell that are not trivial. Again, the driver takes are of this and all Linux sees is a nice compliant config space.
Quoted 6 lines
If that is indeed the case, them I would vote for a slight modification of the intermediate patch that I cited earlier - the one in which the ranges property includes translations from those special config addresses into CPU addresses. The modification is to fix the sizes, changing
0x2000to0x800, so those ranges entries do not overlap in the child address domain.
On Marvell the size of the MMIO space is 0x2000, there are imporant registers there beyond index 0x800 (todays driver may not touch them but HW has them). Reducing the size doesn't seem appropriate.
Quoted 3 lines
pcie-controller {
compatible = "marvell,armada-370-pcie";
status = "disabled";device_type = "pci";
Here as well, as you noted before?
Quoted 12 lines
#address-cells = <3>; #size-cells = <2>; bus-range = <0x00 0xff>; ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */ 0x82000000 0 0xe0000000 0xe0000000 0 0x08000000 /* non-prefetchable memory */ 0x81000000 0 0 0xe8000000 0 0x00100000>; /*
Quoted 5 lines
downstream I/O */
pcie at 1,0 { device_type = "pci"; reg = <0x0800 0 0 0 0>;
Okay, this was Thierry's original idea, and it was my note that this didn't seem like a good choice. I'm not wedded to that, but I'll explain my reasoning a bit.
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonky
- Linux's OF core doesn't parse a 'reg' property under 'device_type = "pci"' as something CPU mappable, it only looks to assigned-addresses for that. So the OF core would need to learn how to handle this case, and it would have to be well defined..
Thierry's original patch avoided this problem by not using device_type = "pci"..
Also, did you mean to have a 0 size for the 'reg'? How will things know how big the MMIO region is? Can't just assume 0x800..
Again, thanks!
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 18 lines · Jason Gunthorpe
Quoted 4 lines · Mitch Bradley
Okay, I think I finally get it. The Marvell root port bridge setup registers look like standard config headers, even though they aren't really in config space (because you need a different access method for them, compared to real config accesses to downstream devices).
Well, thats pretty close. On Marvell the MMIO space has a lot of stuff, some of it is config related, lots of it isn't. In fact most of it isn't. Tegra is different, the MMIO region is exactly the config space of the root port bridge.
Tegra could probably happily and sanely do as you've described (well, see below), but it is wonky for Marvell. It looks like there are more drivers coming that have this need - so I had hoped for a general answer to the 'per-port MMIO space' problem that is unrelated to the MMIO spaces role in config access.
Quoted 8 lines
Quoted 4 lines
So, in order for the common code that enumerates the PCI bus to work, you need to "spoof" the config accesses so that when you try to access those particular "pseudo config headers", it uses the MMIO method instead of the real config access method.
There are rules for config access in both Tegra and Marvell that are not trivial. Again, the driver takes are of this and all Linux sees is a nice compliant config space.
Thanks for explaining that.
So it seems that we are faced with two requirements that are somewhat at odds with one another.
1) Some of the root port bridge registers have to be accessed via config space access functions so common PCI enumeration code will work. To represent this with the usual DT structure, the top root-complex needs to define a 3/2 address space so its children
can have standard PCI reg properties. Presumably, if those registers are being programmed by common code, Marvell-specific code would restrict its role to setting up config-space access functions, leaving the actual touching of the registers to the common code.
2) Marvell chips have additional non-standard per-root-port registers that generic PCI code would not understand. These registers would be touched only by Marvell-specific code.
The two kinds of registers are adjacent in MMIO space. However, unless I am misunderstanding this MV78230 manual, the highest "config header" register index is 0x134, well below the 0x1000 size limit of a PCIe config header. Some of the extra registers are at 0x8xx, and others are above 0x1800.
That suggests the following:
For the "config header" registers that use generic PCI code, use a ranges entry to associate (pass through) MMIO addresses to the config header portion of the register block. This parameterizes the code that sets up the special-case PCI config access. That specification technique is general and could be used not only for the Marvell case, but also for any other chip that has some number of direct-mapped config headers.
For requirement (2), the top node has a reg property listing the portions of the address space that are consumed by the driver at the top level instead of being passed through to the PCI addressing domain. E.g.
reg = <0xd0040800 0x1800>, <0xd0080800 0x1800>;
Thus we have accurately described the two aspects of the true situation.
The PCI-compliant "config header" registers are passed through to the
child nodes where they can be dealt with in the normal PCI fashion. The non-PCI-compliant register footprint lives within the Marvell-specific root-complex driver.
The root-complex driver presumably needs associate non-compliant register blocks with specific child nodes. That can be done by requiring that the reg entries are in the same order as the config-space ranges entries.
Anticipating the possible objection that ARM's 0x1000-byte page size does not permit virtual-to-physical mapping at 0x800-byte granularity:
The device tree does not guarantee that reg entries are page-aligned; it simply tries to describe the reality, even though it might be messy.
Quoted 7 lines
Quoted 7 lines
If that is indeed the case, them I would vote for a slight modification of the intermediate patch that I cited earlier - the one in which the ranges property includes translations from those special config addresses into CPU addresses. The modification is to fix the sizes, changing
0x2000to0x800, so those ranges entries do not overlap in the child address domain.
BTW I have completely changed my mind about the overlap thing.
I said that it was bogus to use size=0x2000 for a config space header.
That was based on an interpretation - which I now dislike - of the
meaning of 3-cell config addresses. By that old interpretation, size=0x800 would also be bogus, because bits 10-8 of the config address are for the function number.
Consider the following question, which I have never previously considered, at least not explicitly:
Q: What would be the 3-cell representation of the Command/Status register address (offset 4) in device 1, function 1?
One obvious - but weak - answer is <0x00000904 0 0> . It's obvious because the value 0x80000904 is what you put in the cf8 indirect access address register. But it is weak because it unnecessarily restricts the config header size, and because it works differently than offsets applied to memory and I/O space addresses. Furthermore, coupling the offset to the cf8 register is dodgy because of the funniness where you have to but the byte-selector bits 1..0 not in the address register cf8, but rather in the data register cfc.
The better answer is <0x00000900 0 4>, using the third cell for the offset, as with IO and memory space offsets.
Under this better interpretation, config space sizes larger than 0x100 are no problem. The first cell is just a selector. The size is "added" to the third cell, so it does not encroach into the first cell's device,function bits.
This better interpretation easily handles PCIe's 0x1000-byte extended config headers, and trivially accomodates even larger sizes should a future PCI variant further increase the header size. as described way down below.
Quoted 4 lines
On Marvell the size of the MMIO space is
0x2000, there are imporant registers there beyond index0x800(todays driver may not touch them but HW has them). Reducing the size doesn't seem appropriate.
Per the above, the current idea is not to reduce the size, but rather to split the 0x2000 into two pieces, accurately reflecting the fact that part of it is PCI-compliant and can be handled by generic PCI drivers, while the rest must be handled differently, outside of standard PCI land. Commingling the two seems dubious.
Quoted 27 lines
Quoted 4 lines
pcie-controller { compatible = "marvell,armada-370-pcie"; status = "disabled";device_type = "pci";Here as well, as you noted before?
Quoted 12 lines
#address-cells = <3>; #size-cells = <2>; bus-range = <0x00 0xff>; ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */ 0x82000000 0 0xe0000000 0xe0000000 0 0x08000000 /* non-prefetchable memory */ 0x81000000 0 0 0xe8000000 0 0x00100000>; /*Quoted 5 lines
downstream I/O */
pcie at 1,0 { device_type = "pci"; reg = <0x0800 0 0 0 0>;
Quoted 7 lines
Okay, this was Thierry's original idea, and it was my note that this didn't seem like a good choice. I'm not wedded to that, but I'll explain my reasoning a bit.
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonky
I agree that mapping config space is sort of a jarring concept, but I think that's because PCs have polluted the mindspace, not because there is anything inherently bad about it. The original PCI spec fundamentally treated config space as just another (rather segmented) linear address space. It specified the cf8/cfc indirect access mechanism not as a deep semantic feature, but rather as an implementation hack to work around addressing limitations of PC chipsets and the addressing-deficient popular OS of the time (Windows 3.1 which was a veneer over DOS). The PCI spec defined configuration mechanism #2 as a direct map into PC I/O space. RISC chipsets of that era (e.g. Alpha, Power PC) often direct-mapped config space into some form of load/store space.
But PCs soon encroached on the RISC market and RISC system vendors started to "leverage" PC I/O chipsets. The indirect access "config mechanism #1" became part of the mindset. Indirect access was hardcoded in enough software that chipset designers would provide indirect access even if direct-mapping was possible.
Fundamentally, a config header is just a block of registers, addressable as an index relative to some base. It might require some specific access path - but that is true with any block of registers. Even direct-mapped MMIO registers often require specific semantics such as non-cached, non-write-buffered, or special address space ids.
The PCI SIG expected PCIe extended config space to be direct-mapped - see slide 25 of
So mappability of config space was intended from the get-go. The real wonky thing is the "indirect access zombie that cannot die".
Quoted 4 lines
- Linux's OF core doesn't parse a 'reg' property under '
device_type= "pci"' as something CPU mappable, it only looks to assigned-addresses for that. So the OF core would need to learn how to handle this case, and it would have to be well defined..
The way I envisioned it working is that the root-complex node defines its own config space access functions (the "ops" argument to pci_scan_root_bus). For config addresses listed in ranges, the functions do MMIO. For others, they use the chipset's indirect-access registers. The OF core is not involved.
Quoted 6 lines
Thierry's original patch avoided this problem by not using
device_type= "pci"..Also, did you mean to have a 0 size for the 'reg'? How will things know how big the MMIO region is? Can't just assume
0x800..
Sorry, that was a typo.
Quoted 4 lines
Again, thanks!
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 123 lines · Mitch Bradley
Quoted 18 lines · Jason Gunthorpe
Quoted 4 lines · Mitch Bradley
Okay, I think I finally get it. The Marvell root port bridge setup registers look like standard config headers, even though they aren't really in config space (because you need a different access method for them, compared to real config accesses to downstream devices).
Well, thats pretty close. On Marvell the MMIO space has a lot of stuff, some of it is config related, lots of it isn't. In fact most of it isn't. Tegra is different, the MMIO region is exactly the config space of the root port bridge.
Tegra could probably happily and sanely do as you've described (well, see below), but it is wonky for Marvell. It looks like there are more drivers coming that have this need - so I had hoped for a general answer to the 'per-port MMIO space' problem that is unrelated to the MMIO spaces role in config access.
Quoted 8 lines
Quoted 4 lines
So, in order for the common code that enumerates the PCI bus to work, you need to "spoof" the config accesses so that when you try to access those particular "pseudo config headers", it uses the MMIO method instead of the real config access method.
There are rules for config access in both Tegra and Marvell that are not trivial. Again, the driver takes are of this and all Linux sees is a nice compliant config space.
Thanks for explaining that.
So it seems that we are faced with two requirements that are somewhat at odds with one another.
1) Some of the root port bridge registers have to be accessed via config space access functions so common PCI enumeration code will work. To represent this with the usual DT structure, the top root-complex needs to define a 3/2 address space so its children
can have standard PCI reg properties. Presumably, if those registers are being programmed by common code, Marvell-specific code would restrict its role to setting up config-space access functions, leaving the actual touching of the registers to the common code.2) Marvell chips have additional non-standard per-root-port registers that generic PCI code would not understand. These registers would be touched only by Marvell-specific code.
The two kinds of registers are adjacent in MMIO space. However, unless I am misunderstanding this MV78230 manual, the highest "config header" register index is
0x134, well below the0x1000size limit of a PCIe config header. Some of the extra registers are at 0x8xx, and others are above0x1800.That suggests the following:
For the "config header" registers that use generic PCI code, use a ranges entry to associate (pass through) MMIO addresses to the config header portion of the register block. This parameterizes the code that sets up the special-case PCI config access. That specification technique is general and could be used not only for the Marvell case, but also for any other chip that has some number of direct-mapped config headers.
For requirement (2), the top node has a reg property listing the portions of the address space that are consumed by the driver at the top level instead of being passed through to the PCI addressing domain. E.g.
reg = <0xd0040800 0x1800>, <0xd0080800 0x1800>;Thus we have accurately described the two aspects of the true situation.
The PCI-compliant "config header" registers are passed through to the
child nodes where they can be dealt with in the normal PCI fashion. The non-PCI-compliant register footprint lives within the Marvell-specific root-complex driver.The root-complex driver presumably needs associate non-compliant register blocks with specific child nodes. That can be done by requiring that the reg entries are in the same order as the config-space ranges entries.
Anticipating the possible objection that ARM's 0x1000-byte page size does not permit virtual-to-physical mapping at 0x800-byte granularity:
The device tree does not guarantee that reg entries are page-aligned; it simply tries to describe the reality, even though it might be messy.Quoted 7 lines
Quoted 7 lines
If that is indeed the case, them I would vote for a slight modification of the intermediate patch that I cited earlier - the one in which the ranges property includes translations from those special config addresses into CPU addresses. The modification is to fix the sizes, changing
0x2000to0x800, so those ranges entries do not overlap in the child address domain.BTW I have completely changed my mind about the overlap thing.
I said that it was bogus to use size=0x2000 for a config space header.
That was based on an interpretation - which I now dislike - of the
meaning of 3-cell config addresses. By that old interpretation, size=0x800 would also be bogus, because bits 10-8 of the config address are for the function number.Consider the following question, which I have never previously considered, at least not explicitly:
Q: What would be the 3-cell representation of the Command/Status register address (offset 4) in device 1, function 1?
One obvious - but weak - answer is <
0x000009040 0> . It's obvious because the value0x80000904is what you put in the cf8 indirect access address register. But it is weak because it unnecessarily restricts the config header size, and because it works differently than offsets applied to memory and I/O space addresses. Furthermore, coupling the offset to the cf8 register is dodgy because of the funniness where you have to but the byte-selector bits 1..0 not in the address register cf8, but rather in the data register cfc.The better answer is <
0x000009000 4>, using the third cell for the offset, as with IO and memory space offsets.Under this better interpretation, config space sizes larger than
0x100are no problem. The first cell is just a selector. The size is "added" to the third cell, so it does not encroach into the first cell's device,function bits.
Quoted 4 lines
This better interpretation easily handles PCIe's 0x1000-byte extended config headers, and trivially accomodates even larger sizes should a future PCI variant further increase the header size. as described way down below.
That makes a lot of sense. It also mirrors parts of a discussion we previously had when we first discussed a DT binding for Tegra PCIe.
Thanks for taking the time to go over this in so much detail. I very much appreciate it.
Quoted 64 lines
Quoted 18 lines
Quoted 12 lines
#address-cells = <3>; #size-cells = <2>; bus-range = <0x00 0xff>; ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */ 0x82000000 0 0xe0000000 0xe0000000 0 0x08000000 /* non-prefetchable memory */ 0x81000000 0 0 0xe8000000 0 0x00100000>; /*Quoted 5 lines
downstream I/O */
pcie at 1,0 { device_type = "pci"; reg = <0x0800 0 0 0 0>;Quoted 7 lines
Okay, this was Thierry's original idea, and it was my note that this didn't seem like a good choice. I'm not wedded to that, but I'll explain my reasoning a bit.
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonkyI agree that mapping config space is sort of a jarring concept, but I think that's because PCs have polluted the mindspace, not because there is anything inherently bad about it. The original PCI spec fundamentally treated config space as just another (rather segmented) linear address space. It specified the cf8/cfc indirect access mechanism not as a deep semantic feature, but rather as an implementation hack to work around addressing limitations of PC chipsets and the addressing-deficient popular OS of the time (Windows 3.1 which was a veneer over DOS). The PCI spec defined configuration mechanism #2 as a direct map into PC I/O space. RISC chipsets of that era (e.g. Alpha, Power PC) often direct-mapped config space into some form of load/store space.
But PCs soon encroached on the RISC market and RISC system vendors started to "leverage" PC I/O chipsets. The indirect access "config mechanism #1" became part of the mindset. Indirect access was hardcoded in enough software that chipset designers would provide indirect access even if direct-mapping was possible.
Fundamentally, a config header is just a block of registers, addressable as an index relative to some base. It might require some specific access path - but that is true with any block of registers. Even direct-mapped MMIO registers often require specific semantics such as non-cached, non-write-buffered, or special address space ids.
The PCI SIG expected PCIe extended config space to be direct-mapped - see slide 25 of
So mappability of config space was intended from the get-go. The real wonky thing is the "indirect access zombie that cannot die".
Quoted 4 lines
- Linux's OF core doesn't parse a 'reg' property under '
device_type= "pci"' as something CPU mappable, it only looks to assigned-addresses for that. So the OF core would need to learn how to handle this case, and it would have to be well defined..
Quoted 9 lines
The way I envisioned it working is that the root-complex node defines its own config space access functions (the "ops" argument to
pci_scan_root_bus). For config addresses listed in ranges, the functions do MMIO. For others, they use the chipset's indirect-access registers. The OF core is not involved.Quoted 3 lines
Thierry's original patch avoided this problem by not using
device_type= "pci"..
I think the OF core still needs to be involved in order to translate the reg = <0x0800 0 0 0 0>; entry into a resource that can be ioremap()'ed. As Jason already said, the OF core only does that for assigned-addresses when device_type = "pci".
Otherwise the pci_ops implementation still couldn't access the MMIO registers. Aiming for a driver-specific solution doesn't seem like a good idea but if the functionality is common enough to be required by two or more drivers perhaps a new helper could be created for exactly this purpose.
Perhaps another alternative would be to extend the OF core to translate the entries in the reg property as well. Or maybe I misunderstood what you said.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130311/4e7dd462/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 209 lines · Thierry Reding
Quoted 123 lines · Mitch Bradley
Quoted 18 lines · Jason Gunthorpe
Quoted 4 lines · Mitch Bradley
Okay, I think I finally get it. The Marvell root port bridge setup registers look like standard config headers, even though they aren't really in config space (because you need a different access method for them, compared to real config accesses to downstream devices).
Well, thats pretty close. On Marvell the MMIO space has a lot of stuff, some of it is config related, lots of it isn't. In fact most of it isn't. Tegra is different, the MMIO region is exactly the config space of the root port bridge.
Tegra could probably happily and sanely do as you've described (well, see below), but it is wonky for Marvell. It looks like there are more drivers coming that have this need - so I had hoped for a general answer to the 'per-port MMIO space' problem that is unrelated to the MMIO spaces role in config access.
Quoted 8 lines
Quoted 4 lines
So, in order for the common code that enumerates the PCI bus to work, you need to "spoof" the config accesses so that when you try to access those particular "pseudo config headers", it uses the MMIO method instead of the real config access method.
There are rules for config access in both Tegra and Marvell that are not trivial. Again, the driver takes are of this and all Linux sees is a nice compliant config space.
Thanks for explaining that.
So it seems that we are faced with two requirements that are somewhat at odds with one another.
1) Some of the root port bridge registers have to be accessed via config space access functions so common PCI enumeration code will work. To represent this with the usual DT structure, the top root-complex needs to define a 3/2 address space so its children
can have standard PCI reg properties. Presumably, if those registers are being programmed by common code, Marvell-specific code would restrict its role to setting up config-space access functions, leaving the actual touching of the registers to the common code.2) Marvell chips have additional non-standard per-root-port registers that generic PCI code would not understand. These registers would be touched only by Marvell-specific code.
The two kinds of registers are adjacent in MMIO space. However, unless I am misunderstanding this MV78230 manual, the highest "config header" register index is
0x134, well below the0x1000size limit of a PCIe config header. Some of the extra registers are at 0x8xx, and others are above0x1800.That suggests the following:
For the "config header" registers that use generic PCI code, use a ranges entry to associate (pass through) MMIO addresses to the config header portion of the register block. This parameterizes the code that sets up the special-case PCI config access. That specification technique is general and could be used not only for the Marvell case, but also for any other chip that has some number of direct-mapped config headers.
For requirement (2), the top node has a reg property listing the portions of the address space that are consumed by the driver at the top level instead of being passed through to the PCI addressing domain. E.g.
reg = <0xd0040800 0x1800>, <0xd0080800 0x1800>;Thus we have accurately described the two aspects of the true situation.
The PCI-compliant "config header" registers are passed through to the
child nodes where they can be dealt with in the normal PCI fashion. The non-PCI-compliant register footprint lives within the Marvell-specific root-complex driver.The root-complex driver presumably needs associate non-compliant register blocks with specific child nodes. That can be done by requiring that the reg entries are in the same order as the config-space ranges entries.
Anticipating the possible objection that ARM's 0x1000-byte page size does not permit virtual-to-physical mapping at 0x800-byte granularity:
The device tree does not guarantee that reg entries are page-aligned; it simply tries to describe the reality, even though it might be messy.Quoted 7 lines
Quoted 7 lines
If that is indeed the case, them I would vote for a slight modification of the intermediate patch that I cited earlier - the one in which the ranges property includes translations from those special config addresses into CPU addresses. The modification is to fix the sizes, changing
0x2000to0x800, so those ranges entries do not overlap in the child address domain.BTW I have completely changed my mind about the overlap thing.
I said that it was bogus to use size=0x2000 for a config space header.
That was based on an interpretation - which I now dislike - of the
meaning of 3-cell config addresses. By that old interpretation, size=0x800 would also be bogus, because bits 10-8 of the config address are for the function number.Consider the following question, which I have never previously considered, at least not explicitly:
Q: What would be the 3-cell representation of the Command/Status register address (offset 4) in device 1, function 1?
One obvious - but weak - answer is <
0x000009040 0> . It's obvious because the value0x80000904is what you put in the cf8 indirect access address register. But it is weak because it unnecessarily restricts the config header size, and because it works differently than offsets applied to memory and I/O space addresses. Furthermore, coupling the offset to the cf8 register is dodgy because of the funniness where you have to but the byte-selector bits 1..0 not in the address register cf8, but rather in the data register cfc.The better answer is <
0x000009000 4>, using the third cell for the offset, as with IO and memory space offsets.Under this better interpretation, config space sizes larger than
0x100are no problem. The first cell is just a selector. The size is "added" to the third cell, so it does not encroach into the first cell's device,function bits.Quoted 4 lines
This better interpretation easily handles PCIe's 0x1000-byte extended config headers, and trivially accomodates even larger sizes should a future PCI variant further increase the header size. as described way down below.
That makes a lot of sense. It also mirrors parts of a discussion we previously had when we first discussed a DT binding for Tegra PCIe.
Thanks for taking the time to go over this in so much detail. I very much appreciate it.
Quoted 64 lines
Quoted 18 lines
Quoted 12 lines
#address-cells = <3>; #size-cells = <2>; bus-range = <0x00 0xff>; ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */ 0x82000000 0 0xe0000000 0xe0000000 0 0x08000000 /* non-prefetchable memory */ 0x81000000 0 0 0xe8000000 0 0x00100000>; /*Quoted 5 lines
downstream I/O */
pcie at 1,0 { device_type = "pci"; reg = <0x0800 0 0 0 0>;Quoted 7 lines
Okay, this was Thierry's original idea, and it was my note that this didn't seem like a good choice. I'm not wedded to that, but I'll explain my reasoning a bit.
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonkyI agree that mapping config space is sort of a jarring concept, but I think that's because PCs have polluted the mindspace, not because there is anything inherently bad about it. The original PCI spec fundamentally treated config space as just another (rather segmented) linear address space. It specified the cf8/cfc indirect access mechanism not as a deep semantic feature, but rather as an implementation hack to work around addressing limitations of PC chipsets and the addressing-deficient popular OS of the time (Windows 3.1 which was a veneer over DOS). The PCI spec defined configuration mechanism #2 as a direct map into PC I/O space. RISC chipsets of that era (e.g. Alpha, Power PC) often direct-mapped config space into some form of load/store space.
But PCs soon encroached on the RISC market and RISC system vendors started to "leverage" PC I/O chipsets. The indirect access "config mechanism #1" became part of the mindset. Indirect access was hardcoded in enough software that chipset designers would provide indirect access even if direct-mapping was possible.
Fundamentally, a config header is just a block of registers, addressable as an index relative to some base. It might require some specific access path - but that is true with any block of registers. Even direct-mapped MMIO registers often require specific semantics such as non-cached, non-write-buffered, or special address space ids.
The PCI SIG expected PCIe extended config space to be direct-mapped - see slide 25 of
So mappability of config space was intended from the get-go. The real wonky thing is the "indirect access zombie that cannot die".
Quoted 4 lines
- Linux's OF core doesn't parse a 'reg' property under '
device_type= "pci"' as something CPU mappable, it only looks to assigned-addresses for that. So the OF core would need to learn how to handle this case, and it would have to be well defined..Quoted 9 lines
The way I envisioned it working is that the root-complex node defines its own config space access functions (the "ops" argument to
pci_scan_root_bus). For config addresses listed in ranges, the functions do MMIO. For others, they use the chipset's indirect-access registers. The OF core is not involved.Quoted 3 lines
Thierry's original patch avoided this problem by not using
device_type= "pci"..
Quoted 7 lines
I think the OF core still needs to be involved in order to translate the reg = <
0x08000 0 0 0>; entry into a resource that can be ioremap()'ed. As Jason already said, the OF core only does that for assigned-addresses whendevice_type= "pci".Otherwise the
pci_opsimplementation still couldn't access the MMIO registers.
I'm not sure I understand your point that <0x0800 ...> needs to be translated into a resource.
(a) Existing code that needs to match a child nodes to a discovered PCI hardware device uses the existing of_pci_find_child_device(parent, devfn) from of_pci.c . That routine "just works", because the reg = <0x0800 ..> entry is standard.
(b) The discovery/enumeration code needs to access config space via pci_ops. The root complex driver implements pci_ops based on a trivial parsing of ranges (omitting irrelevant details):
pci_op_read(devfn, pos) {
loop_over_ranges_entries {
if (to_devfn(ranges_entry.child) == devfn) {
return mmio_read(ranges_entry.parent + pos); }
return indirect_read(devfn, pos);
}
The actual implementation might setup different data structures, but the pseudocode captures the essential points: The input argument is devfn, and a simple scan of ranges controls whether to use MMIO or indirect access for a given devfn value. Every aspect of the scanning loop is trivial. The loop itself is just a stride-5 walk over an array of integers.
"ranges_entry.parent" is be32_to_cpu(ranges_entry[3])
and
"to_devfn(ranges_entry.child)" is "be32_to_cpup(ranges_entry[0]) >> 8"
(you don't want to mask the result in this case because not masking prevents non-config-space ranges entries from matching devfn.)
Quoted 4 lines
Aiming for a driver-specific solution doesn't seem like a good idea but if the functionality is common enough to be required by two or more drivers perhaps a new helper could be created for exactly this purpose.
Indeed - but it is often best to wait and see rather than guessing about what sort of common helper is needed. I usually guess wrong in such cases.
Quoted 7 lines
Perhaps another alternative would be to extend the OF core to translate the entries in the reg property as well. Or maybe I misunderstood what you said.
Thierry
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 8 lines
(b) The discovery/enumeration code needs to access config space via
pci_ops. The root complex driver implementspci_opsbased on a trivial parsing of ranges (omitting irrelevant details):pci_op_read(devfn, pos) { loop_over_ranges_entries { if (to_devfn(ranges_entry.child) == devfn) { return mmio_read(ranges_entry.parent + pos);
^^^^^^^^^^^^^^^^^^^
This has to be converted through all enclosing node's ranges prior to being used as a CPU address - the OF core has all the code to do this, but a new entry point would be needed for this specific application...
Cheers,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 8 lines · Jason Gunthorpe
Quoted 8 lines
(b) The discovery/enumeration code needs to access config space via
pci_ops. The root complex driver implementspci_opsbased on a trivial parsing of ranges (omitting irrelevant details):pci_op_read(devfn, pos) { loop_over_ranges_entries { if (to_devfn(ranges_entry.child) == devfn) { return mmio_read(ranges_entry.parent + pos);
Quoted 5 lines
^^^^^^^^^^^^^^^^^^^This has to be converted through all enclosing node's ranges prior to being used as a CPU address - the OF core has all the code to do this, but a new entry point would be needed for this specific application...
Agreed. But I was considering that detail to be outside the scope of this discussion because the same thing must happen for any physical address at this level of the tree.
Quoted 4 lines
Cheers,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Mitch BradleyQuoted 16 lines · Mitch Bradley
So it seems that we are faced with two requirements that are somewhat at odds with one another.
1) Some of the root port bridge registers have to be accessed via config space access functions so common PCI enumeration code will work. To represent this with the usual DT structure, the top root-complex needs to define a 3/2 address space so its children
can have standard PCI reg properties. Presumably, if those registers are being programmed by common code, Marvell-specific code would restrict its role to setting up config-space access functions, leaving the actual touching of the registers to the common code.2) Marvell chips have additional non-standard per-root-port registers that generic PCI code would not understand. These registers would be touched only by Marvell-specific code.
Quoted 5 lines
The two kinds of registers are adjacent in MMIO space. However, unless I am misunderstanding this MV78230 manual, the highest "config header" register index is
0x134, well below the0x1000size limit of a PCIe config header. Some of the extra registers are at 0x8xx, and others are above0x1800.
The register block you are looking at is for the cores 'target end port mode', and the MMIO version is for internal use only, to allow the CPU to configure RO bits, and other tasks. The target core will allow access to that space via config cycles, either internally (through the indirection register) or externally (from the off-chip root complex) generated.
However - the driver runs the core in a 'root port bridge mode' where the config header register block you are looking at is inhibited. The Marvell IP block requires software support to run in bridge mode. So Marvell really has only (2), while Tegra has only (1).
Quoted 5 lines
For requirement (2), the top node has a reg property listing the portions of the address space that are consumed by the driver at the top level instead of being passed through to the PCI addressing domain. E.g.
reg = <0xd0040800 0x1800>, <0xd0080800 0x1800>;
Okay, so this is agreeing with what Thomas already has:
http://www.spinics.net/lists/arm-kernel/msg228749.html
With your two notes:
- Correct the pcie at x,y to match the OF spec
- Add a 'device_type = "pci"' to the pcie-controller node
Is that right?
Quoted 5 lines
I said that it was bogus to use size=0x2000 for a config space header.
That was based on an interpretation - which I now dislike - of the
meaning of 3-cell config addresses. By that old interpretation, size=0x800 would also be bogus, because bits 10-8 of the config address are for the function number.
Hum, I thought the spec was pretty clear on this point:
00 denotes Configuration Space, in which case:
[..]
bbbbbbbb,ddddd,fff,rrrrrrrrr is the Configuration Space address hh...hh,ll...ll must be zero
And also the text at 2.1.4.4..
So you get an 8 bit register offset, placed in the highest DW..
Can you read things another way?
Is there an updated spec that supports PCI-E extended config space?
Quoted 5 lines
Consider the following question, which I have never previously considered, at least not explicitly:
Q: What would be the 3-cell representation of the Command/Status register address (offset 4) in device 1, function 1?
Well, see section 11.1.2, where it provides an example for the 'Expansion ROM base address register (0x30)' as being:
02xxxx30 00000000 00000000
Also the f-code bindings say:
The data type 'config-addr' refers to the 'phys.hi' cell of the numerical representation of a Configuration Space address.
So there is an strong convention to use the 'r' bits as the offset..
Further, review section 12 about how ranges should be treated - it specifically says that the b,d,f bits in ranges should be 0, and the child address should have those bits masked prior to searching the ranges.
Section 12 would seem to forbid this:
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */
0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */Are you reading that differently?
Quoted 7 lines
Quoted 3 lines
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonkyI agree that mapping config space is sort of a jarring concept, but I think that's because PCs have polluted the mindspace, not because there
I'm well aware of MMIO config space acces, that isn't so jarring. What is so jarring is the idea that the OF translation would work like:
<0x00000900 0 0> -> 0xd4004000
<0x00000901 0 0> -> 0xd4004001
Which is completely unlike any ranges translation I've ever seen.
Basically, I look at how the register offset is encoded in the higest dword plus the statement in section 12, and and conclude the there wasn't an intention to model a memory map'd config space through OF.
What am I missing here?
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 31 lines · Jason Gunthorpe
Quoted 16 lines · Mitch Bradley
So it seems that we are faced with two requirements that are somewhat at odds with one another.
1) Some of the root port bridge registers have to be accessed via config space access functions so common PCI enumeration code will work. To represent this with the usual DT structure, the top root-complex needs to define a 3/2 address space so its children
can have standard PCI reg properties. Presumably, if those registers are being programmed by common code, Marvell-specific code would restrict its role to setting up config-space access functions, leaving the actual touching of the registers to the common code.2) Marvell chips have additional non-standard per-root-port registers that generic PCI code would not understand. These registers would be touched only by Marvell-specific code.
Quoted 5 lines
The two kinds of registers are adjacent in MMIO space. However, unless I am misunderstanding this MV78230 manual, the highest "config header" register index is
0x134, well below the0x1000size limit of a PCIe config header. Some of the extra registers are at 0x8xx, and others are above0x1800.The register block you are looking at is for the cores 'target end port mode', and the MMIO version is for internal use only, to allow the CPU to configure RO bits, and other tasks. The target core will allow access to that space via config cycles, either internally (through the indirection register) or externally (from the off-chip root complex) generated.
Quoted 4 lines
However - the driver runs the core in a 'root port bridge mode' where the config header register block you are looking at is inhibited. The Marvell IP block requires software support to run in bridge mode. So Marvell really has only (2), while Tegra has only (1).
Interesting.
For Marvell, if Marvell has only (2), does that imply that standard PCI discovery and enumeration code cannot find the root port bridges, and also there is no way to auto-configure the bridges with common pci-pci bridge code?
Quoted 9 lines
Quoted 6 lines
For requirement (2), the top node has a reg property listing the portions of the address space that are consumed by the driver at the top level instead of being passed through to the PCI addressing domain. E.g.
reg = <0xd0040800 0x1800>, <0xd0080800 0x1800>;Okay, so this is agreeing with what Thomas already has:
Quoted 7 lines
http://www.spinics.net/lists/arm-kernel/msg228749.html
With your two notes:
- Correct the pcie at x,y to match the OF spec
- Add a 'device_type= "pci"' to the pcie-controller nodeIs that right?
At this point I am confused again. There was talk of the need to use standard PCI enumeration code to deal with the bridges, thus the need for 3/2 to describe the interface between root-complex and the child root-port nodes. But now I think I'm hearing that these particular root ports bear no resemblance to standard pcie bridges. So I am back to "not sure why we are using 3/2 PCI addresses across an interface boundary that has no relation to PCI addressing". Sorry if I'm being obtuse.
For Tegra, I think it all makes sense, and I still like the ranges representation to control the config-space spoofing. But for Marvell, it seems like there is neither the need for nor the possibility of spoofing, nor for PCI-style addressing at the complex/port boundary.
Quoted 14 lines
Quoted 6 lines
I said that it was bogus to use size=0x2000 for a config space header.
That was based on an interpretation - which I now dislike - of the
meaning of 3-cell config addresses. By that old interpretation, size=0x800 would also be bogus, because bits 10-8 of the config address are for the function number.Hum, I thought the spec was pretty clear on this point:
00 denotes Configuration Space, in which case:
[..]
bbbbbbbb,ddddd,fff,rrrrrrrrr is the Configuration Space address hh...hh,ll...ll must be zero
Quoted 5 lines
And also the text at 2.1.4.4..
So you get an 8 bit register offset, placed in the highest DW..
Can you read things another way?
The reading is clear, but there is a crucial point that is missing.
I don't recall ever having seen anything but 0 in rrrrrrrr. PCI reg properties always list the base address of config header. There is never a need for a reg entry to point into the middle of a config header. The granularity is always at the dev,function level. So the fact that the spec puts the offset in those bits is moot, because the offset is never used. (I will address your counterargument below.)
I didn't anticipate that fact when I developed the representation, nor did I anticipate that the rather-rigidly structured config space would be extended years later.
With my "new interpretation", you still never actually set any of the "offset bits". The only thing you get from it is permission to use larger size values without creating algebraic nonsense. You never actually do arithmetic on that representation.
Quoted 2 lines
Is there an updated spec that supports PCI-E extended config space?
http://www.openfirmware.org/ofwg/bindings/pci/pci-express.txt
It puts the extended config address bits in 27:24 of the high word - but that said, I doubt that those bits have ever been set to nonzero. Again because you never need to point to an individual register using this representation. I will call David Kahn, who lives down the street from me, later today and see what he thinks.
Quoted 12 lines
Quoted 6 lines
Consider the following question, which I have never previously considered, at least not explicitly:
Q: What would be the 3-cell representation of the Command/Status register address (offset 4) in device 1, function 1?
Well, see section 11.1.2, where it provides an example for the 'Expansion ROM base address register (
0x30)' as being:02xxxx30 00000000 00000000
Quoted 7 lines
Also the f-code bindings say:
The data type 'config-addr' refers to the 'phys.hi' cell of the numerical representation of a Configuration Space address.
So there is an strong convention to use the 'r' bits as the offset..
You're right, the spec does indeed say that. But in practice, it never actually happened that way. Expansion ROMs, in practice, had to be copied into memory because the tended to disappear under certain settings of various enable bits. And even that copying had to be done very carefully, because expansion ROM access had an annoying tendency to hang some bus interfaces.
So, yeah, the spec implies the convention, but I don't think it was ever used.
Quoted 6 lines
Further, review section 12 about how ranges should be treated - it specifically says that the b,d,f bits in ranges should be 0, and the child address should have those bits masked prior to searching the ranges.
Quoted 6 lines
Section 12 would seem to forbid this:
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */Are you reading that differently?
I wasn't reading it at all, until you pointed it out to me :-) I was just reasoning based on my mental model.
Again, I agree with your reading of the spec, but I can't think of any reason why relaxing that restriction to permit config space mappings would be a problem.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Two things bothered me
- Describing a CPU MMIO mapping with a config address space seems wonkyI agree that mapping config space is sort of a jarring concept, but I think that's because PCs have polluted the mindspace, not because there
Quoted 7 lines
I'm well aware of MMIO config space acces, that isn't so jarring. What is so jarring is the idea that the OF translation would work like:
<
0x000009000 0> ->0xd4004000
<0x000009010 0> ->0xd4004001Which is completely unlike any ranges translation I've ever seen.
You never actually say <0x901 0 0> in the DT. What you say "there is a config header based at <0x900 0 0>". The offset is maintained separately, not added directly to the DT representation of the config header base address.
The most useful interpretation of a ranges entry is that the tuples (child_base,offset) and (parent_base,offset) correspond for 0 <= offset < size. In some cases, you might be able to use arithmetic to merge base and offset; in other cases you can't.
Quoted 6 lines
Basically, I look at how the register offset is encoded in the higest dword plus the statement in section 12, and and conclude the there wasn't an intention to model a memory map'd config space through OF.
What am I missing here?
The real point of section 12 is that you have to attend to the fact that PCI a structured address space, not just a single linear space.
As you say, the text does not convey the intention of mapping config space, but I can say categorically that I certainly did not intent "thou shalt not map config space".
Quoted 4 lines
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 11 lines · Mitch Bradley
Quoted 4 lines
However - the driver runs the core in a 'root port bridge mode' where the config header register block you are looking at is inhibited. The Marvell IP block requires software support to run in bridge mode. So Marvell really has only (2), while Tegra has only (1).
Interesting.
For Marvell, if Marvell has only (2), does that imply that standard PCI discovery and enumeration code cannot find the root port bridges, and also there is no way to auto-configure the bridges with common pci-pci bridge code?
The driver is required to take care of all of this. From the common code's perspective there is a bridge config header, and writing to it controls the device, as the PCI spec intends. But the driver doesn't copy those read/writes 1:1 to any HW register block, it distributes them around the device's register space as necessary to create the PCI bridge config header.
The DT is still modeling the HW, there really are root port bridge(s), that are completely conformant at the TLP level, but the IP designers choose to leave the detail of the bridge config space up to a SW implementation.
Quoted 4 lines
At this point I am confused again. There was talk of the need to use standard PCI enumeration code to deal with the bridges, thus the need for 3/2 to describe the interface between root-complex and the child root-port nodes.
I think your confusion is coming from your expectation that the PCI-E IP block would be an entirely self contained root port, rather than 'tools to build a root port, with the right driver SW'.
Quoted 10 lines
Quoted 10 lines
Further, review section 12 about how ranges should be treated - it specifically says that the b,d,f bits in ranges should be 0, and the child address should have those bits masked prior to searching the ranges.
Section 12 would seem to forbid this:
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */
Quoted 8 lines
Quoted 1 line
Are you reading that differently?
I wasn't reading it at all, until you pointed it out to me :-) I was just reasoning based on my mental model.
Again, I agree with your reading of the spec, but I can't think of any reason why relaxing that restriction to permit config space mappings would be a problem.
Okay, just wanted to confirm that, I think I follow you now. :)
So you are basically proposing a small update to Section 12 that would read something like:
'When matching a config space address, the b,d,f bits are valid in the
ranges and left unmasked in the child. The r bits in the ranges should be 0, and the r bits from the child are copied in the last DW and masked off in the first DW.'
Which, as you note, would not conflict with any existing usage..
Though, I can already see that the above language is not good enough, it doesn't elegantly represent standard ECAM, for instance.. How would you even represent ECAM with ranges??
Another wrinkle is that tegra not only has the per-bridge memory mapped config space, bit it also has 4 'all port' ECAM-like memory mapped config spaces. The rules around them are sufficiently specialized that I'm not sure a ranges mapping would ever be practical.. So right out of the gate we'd fail to capture memory mapped config access via ranges on at least one SOC.
So, to me, this seems like overkill. Only ECAM has a hope of having a common implementation, everything else will be host driver specific, so why go to the trouble to revise the OF PCI specification for config space?
Particularly since there are patches waiting on this question being resolved :)
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 21 lines · Jason Gunthorpe
Quoted 11 lines · Mitch Bradley
Quoted 4 lines
However - the driver runs the core in a 'root port bridge mode' where the config header register block you are looking at is inhibited. The Marvell IP block requires software support to run in bridge mode. So Marvell really has only (2), while Tegra has only (1).
Interesting.
For Marvell, if Marvell has only (2), does that imply that standard PCI discovery and enumeration code cannot find the root port bridges, and also there is no way to auto-configure the bridges with common pci-pci bridge code?
The driver is required to take care of all of this. From the common code's perspective there is a bridge config header, and writing to it controls the device, as the PCI spec intends. But the driver doesn't copy those read/writes 1:1 to any HW register block, it distributes them around the device's register space as necessary to create the PCI bridge config header.
Quoted 4 lines
The DT is still modeling the HW, there really are root port bridge(s), that are completely conformant at the TLP level, but the IP designers choose to leave the detail of the bridge config space up to a SW implementation.
Oh, ouch, deep spoofing. I had to do that in system management mode firmware for a Geode chip. What a pain that was.
Quoted 52 lines
Quoted 5 lines
At this point I am confused again. There was talk of the need to use standard PCI enumeration code to deal with the bridges, thus the need for 3/2 to describe the interface between root-complex and the child root-port nodes.
I think your confusion is coming from your expectation that the PCI-E IP block would be an entirely self contained root port, rather than 'tools to build a root port, with the right driver SW'.
Quoted 10 lines
Quoted 10 lines
Further, review section 12 about how ranges should be treated - it specifically says that the b,d,f bits in ranges should be 0, and the child address should have those bits masked prior to searching the ranges.
Section 12 would seem to forbid this:
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */Quoted 8 lines
Quoted 1 line
Are you reading that differently?
I wasn't reading it at all, until you pointed it out to me :-) I was just reasoning based on my mental model.
Again, I agree with your reading of the spec, but I can't think of any reason why relaxing that restriction to permit config space mappings would be a problem.
Okay, just wanted to confirm that, I think I follow you now. :)
So you are basically proposing a small update to Section 12 that would read something like:
'When matching a config space address, the b,d,f bits are valid in the
ranges and left unmasked in the child. The r bits in the ranges should be 0, and the r bits from the child are copied in the last DW and masked off in the first DW.'Which, as you note, would not conflict with any existing usage..
Though, I can already see that the above language is not good enough, it doesn't elegantly represent standard ECAM, for instance.. How would you even represent ECAM with ranges??
Another wrinkle is that tegra not only has the per-bridge memory mapped config space, bit it also has 4 'all port' ECAM-like memory mapped config spaces. The rules around them are sufficiently specialized that I'm not sure a ranges mapping would ever be practical.. So right out of the gate we'd fail to capture memory mapped config access via ranges on at least one SOC.
Quoted 7 lines
So, to me, this seems like overkill. Only ECAM has a hope of having a common implementation, everything else will be host driver specific, so why go to the trouble to revise the OF PCI specification for config space?
Particularly since there are patches waiting on this question being resolved :)
Okay, so I'll just bow out. Do what you have to do.
Quoted 4 lines
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 87 lines · Mitch Bradley
Quoted 21 lines · Jason Gunthorpe
Quoted 11 lines · Mitch Bradley
Quoted 4 lines
However - the driver runs the core in a 'root port bridge mode' where the config header register block you are looking at is inhibited. The Marvell IP block requires software support to run in bridge mode. So Marvell really has only (2), while Tegra has only (1).
Interesting.
For Marvell, if Marvell has only (2), does that imply that standard PCI discovery and enumeration code cannot find the root port bridges, and also there is no way to auto-configure the bridges with common pci-pci bridge code?
The driver is required to take care of all of this. From the common code's perspective there is a bridge config header, and writing to it controls the device, as the PCI spec intends. But the driver doesn't copy those read/writes 1:1 to any HW register block, it distributes them around the device's register space as necessary to create the PCI bridge config header.
Quoted 4 lines
The DT is still modeling the HW, there really are root port bridge(s), that are completely conformant at the TLP level, but the IP designers choose to leave the detail of the bridge config space up to a SW implementation.
Oh, ouch, deep spoofing. I had to do that in system management mode firmware for a Geode chip. What a pain that was.
Quoted 57 lines
Quoted 5 lines
At this point I am confused again. There was talk of the need to use standard PCI enumeration code to deal with the bridges, thus the need for 3/2 to describe the interface between root-complex and the child root-port nodes.
I think your confusion is coming from your expectation that the PCI-E IP block would be an entirely self contained root port, rather than 'tools to build a root port, with the right driver SW'.
Quoted 10 lines
Quoted 10 lines
Further, review section 12 about how ranges should be treated - it specifically says that the b,d,f bits in ranges should be 0, and the child address should have those bits masked prior to searching the ranges.
Section 12 would seem to forbid this:
ranges = <0x00000800 0 0 0xd4004000 0 0x00000800 /* Root port config header */ 0x00001000 0 0 0xd4008000 0 0x00000800 /* Root port config header */Quoted 8 lines
Quoted 1 line
Are you reading that differently?
I wasn't reading it at all, until you pointed it out to me :-) I was just reasoning based on my mental model.
Again, I agree with your reading of the spec, but I can't think of any reason why relaxing that restriction to permit config space mappings would be a problem.
Okay, just wanted to confirm that, I think I follow you now. :)
So you are basically proposing a small update to Section 12 that would read something like:
'When matching a config space address, the b,d,f bits are valid in the
ranges and left unmasked in the child. The r bits in the ranges should be 0, and the r bits from the child are copied in the last DW and masked off in the first DW.'Which, as you note, would not conflict with any existing usage..
Though, I can already see that the above language is not good enough, it doesn't elegantly represent standard ECAM, for instance.. How would you even represent ECAM with ranges??
Another wrinkle is that tegra not only has the per-bridge memory mapped config space, bit it also has 4 'all port' ECAM-like memory mapped config spaces. The rules around them are sufficiently specialized that I'm not sure a ranges mapping would ever be practical.. So right out of the gate we'd fail to capture memory mapped config access via ranges on at least one SOC.
So, to me, this seems like overkill. Only ECAM has a hope of having a common implementation, everything else will be host driver specific, so why go to the trouble to revise the OF PCI specification for config space?
Quoted 4 lines
Quoted 2 lines
Particularly since there are patches waiting on this question being resolved :)
Okay, so I'll just bow out. Do what you have to do.
So to recapitulate, we agree that configuration space can be translated through the "ranges" property. That means the only missing link is a new function to translate not only "assigned-addresses" but also the "reg" property for PCI devices. Is that it?
And for Marvell the non-configuration-space per-root-port registers need to be mapped separately from the configuration space windows?
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130312/1908ebed/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Thierry Reding
So to recapitulate, we agree that configuration space can be translated through the "ranges" property. That means the only missing link is a new function to translate not only "assigned-addresses" but also the "reg" property for PCI devices. Is that it?
No, the first conclusion is that placing a config space in ranges is against the language in the current spec (see section 12).
The second conclusion is that there is probably a way forward to update the spec in a backwards compatible way to model ECAM, but would require more analysis.
Finally, there was no objections to the approach in Thomas's patch, other than the note to fix the '@1,0' and the extra device_type="pci" ..
Quoted 2 lines
And for Marvell the non-configuration-space per-root-port registers need to be mapped separately from the configuration space windows?
No, there are no configuration space windows on Marvell, so only one mapping.
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Jason Gunthorpe
Quoted 4 lines · Thierry Reding
So to recapitulate, we agree that configuration space can be translated through the "ranges" property. That means the only missing link is a new function to translate not only "assigned-addresses" but also the "reg" property for PCI devices. Is that it?
No, the first conclusion is that placing a config space in ranges is against the language in the current spec (see section 12).
Quoted 6 lines
The second conclusion is that there is probably a way forward to update the spec in a backwards compatible way to model ECAM, but would require more analysis.
Finally, there was no objections to the approach in Thomas's patch, other than the note to fix the '@1,0' and the extra device_type="pci"
That's not what I deduced from Mitch's comments. He said explicitly that he liked the idea to represent mapped configuration space using the ranges property. Mitch, correct me if I misinterpret what you said.
I've also verified that things actually do work with the binding that I proposed for Tegra, even if I add device_type = "pci" to the root port nodes. The reason is that the OF core looks up the type of the *parent* node to find a matching bus. This happens to be the pcie-controller node which is not a PCI device and therefore the address translation works properly.
So let me quote the latest binding that I'm using:
pcie-controller {
compatible = "nvidia,tegra20-pcie";
reg = <0x80003000 0x00000800 /* PADS registers */
0x80003800 0x00000200 /* AFI registers */
0x90000000 0x10000000>; /* configuration space */
reg-names = "pads", "afi", "cs";
interrupts = <0 98 0x04 /* controller interrupt */
0 99 0x04>; /* MSI interrupt */
interrupt-names = "intr", "msi";
bus-range = <0x00 0xff>;
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x00000800 0 0 0x80000000 0 0x00001000 /* port 0 configuration space */
0x00001000 0 0 0x80001000 0 0x00001000 /* port 1 configuration space */
0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */
0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
clocks = <&tegra_car 70>, <&tegra_car 72>, <&tegra_car 74>,
<&tegra_car 118>;
clock-names = "pex", "afi", "pcie_xclk", "pll_e";
status = "disabled";
pci at 1,0 {
device_type = "pci";
reg = <0x000800 0 0 0 0x1000>;
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
nvidia,num-lanes = <2>;
};
pci at 2,0 {
device_type = "pci";
reg = <0x001000 0 0 0 0x1000>;
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
nvidia,num-lanes = <2>;
};
};I can't find anything wrong in the above. As far as I can tell, there's nothing in it that's in conflict with the specification.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130312/c5981d10/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 6 lines · Thierry Reding
I've also verified that things actually do work with the binding that I proposed for Tegra, even if I add
device_type= "pci" to the root port nodes. The reason is that the OF core looks up the type of the *parent* node to find a matching bus. This happens to be the pcie-controller node which is not a PCI device and therefore the address translation works properly.
Mitch pointed out early on that the parent node needs a 'device_type = "pci"' as well.
The pcie-controller node would be classified as a 'bus node' according to the spec.
Not going down the of_pci_* code paths for address translation at the root port bridge nodes is certainly not right.
Quoted 2 lines
ranges = <
0x000008000 00x8000000000x00001000/* port 0 configuration space */0x00001000 0 0 0x80001000 0 0x00001000 /* port 1 configuration space */
Section 12 says that the 'b,d,f' bits are 0 in ranges, so the above is in conflict with that language:
In particular, the phys.hi fields of the child address spaces in the "ranges" property for PCI does not contain the same information as "reg" property entries within PCI nodes. The only information that is present in "ranges" phys.hi entries are the non-relocatable, prefetchable and the PCI address space bits for which the entry applies. I.e., only the n, p and ss bits are present; the bbbbbbbb, ddddd, fff and rrrrrrrr fields are 0.
And the further requirement:
When an address is to be mapped through a PCI bus bridge node, the phys.hi value of the address to be mapped and the child field of a "ranges" entry should be masked so that only the ss bits are compared.
Which is embodied by the code in of_bus_pci_map, meaning the OF core won't even look at the bits in the high DW when searching ranges.
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Jason Gunthorpe
Quoted 6 lines · Thierry Reding
I've also verified that things actually do work with the binding that I proposed for Tegra, even if I add
device_type= "pci" to the root port nodes. The reason is that the OF core looks up the type of the *parent* node to find a matching bus. This happens to be the pcie-controller node which is not a PCI device and therefore the address translation works properly.
Quoted 8 lines
Mitch pointed out early on that the parent node needs a '
device_type= "pci"' as well.The pcie-controller node would be classified as a 'bus node' according to the spec.
Not going down the of_pci_* code paths for address translation at the root port bridge nodes is certainly not right.
I'm not so sure. Why should the pcie-controller be a PCI device? It really isn't. Instead it is a device on the processor/SoC/platform bus that bridges to the PCI bus. As such its children should be PCI devices to reflect that the translation happens within the pcie-controller node.
In the above it is logical to not set the pcie-controller's device_type property to "pci".
Quoted 14 lines
Quoted 2 lines
ranges = <
0x000008000 00x8000000000x00001000/* port 0 configuration space */0x00001000 0 0 0x80001000 0 0x00001000 /* port 1 configuration space */Section 12 says that the 'b,d,f' bits are 0 in ranges, so the above is in conflict with that language:
In particular, the phys.hi fields of the child address spaces in the "ranges" property for PCI does not contain the same information as "reg" property entries within PCI nodes. The only information that is present in "ranges" phys.hi entries are the non-relocatable, prefetchable and the PCI address space bits for which the entry applies. I.e., only the n, p and ss bits are present; the bbbbbbbb, ddddd, fff and rrrrrrrr fields are 0.
Quoted 8 lines
And the further requirement:
When an address is to be mapped through a PCI bus bridge node, the phys.hi value of the address to be mapped and the child field of a "ranges" entry should be masked so that only the ss bits are compared.
Which is embodied by the code in
of_bus_pci_map, meaning the OF core won't even look at the bits in the high DW when searching ranges.
Okay, so the problem with that is that we won't be able to include more entries for configuration space within a ranges property because we can't uniquely map them (they both have the same ss field values). One possibility would be to differentiate using the phys.mid and phys.low cells but they need to be 0 according to the spec.
The other alternative would be to amend the specification. Besides the fact that the specification says so I don't see any reason why this shouldn't be allowed.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130312/ca54ca56/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Thierry Reding
Quoted 2 lines
Not going down the of_pci_* code paths for address translation at the root port bridge nodes is certainly not right.
I'm not so sure. Why should the pcie-controller be a PCI device?
The spec is clear on that point as well:
device_type
A Standard Package conforming to this specification and corresponding to a device that implements a PCI bus shall implement this property with the string value "pci"
The children of a pcie-controller node are PCI devices, thus the pcie-controller node 'implements a PCI bus'. Or, as you say 'device on the processor/SoC/platform bus that bridges to the PCI bus'
Think of device_type as also meaning bus_type and it makes more logical sense, the name is terrible, but its usage is governed by the OF docs..
Quoted 3 lines
The other alternative would be to amend the specification. Besides the fact that the specification says so I don't see any reason why this shouldn't be allowed.
'the specification says so' *IS* the reason. DT isn't a free for all where you get to do whatever you want, or whatever 'feels' right. It is supposed to be a stable, OS agnostic ABI. That means bindings have to follow the specs (when available).
Maybe a future revision will support PCI-E ECAM, but we don't know what that will look like, and I'm pretty sure you don't want to hold up your patches until an IEEE committee gets around to deciding something ;)
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Jason Gunthorpe
Quoted 4 lines · Thierry Reding
Quoted 2 lines
Not going down the of_pci_* code paths for address translation at the root port bridge nodes is certainly not right.
I'm not so sure. Why should the pcie-controller be a PCI device?
The spec is clear on that point as well:
device_type
A Standard Package conforming to this specification and corresponding to a device that implements a PCI bus shall implement this property with the string value "pci"
Quoted 7 lines
The children of a pcie-controller node are PCI devices, thus the pcie-controller node 'implements a PCI bus'. Or, as you say 'device on the processor/SoC/platform bus that bridges to the PCI bus'
Think of
device_typeas also meaningbus_typeand it makes more logical sense, the name is terrible, but its usage is governed by the OF docs..
I beg to differ about the name. device_type means "what kind of functionality is implemented by this device", not "what interface
connects this device to the rest of the system". The use of
"device_type" for that meaning is perfectly reasonable in the English language.
The DT answers the "what interface connects me to the system" question by looking at the parent.
Quoted 10 lines
Quoted 4 lines
The other alternative would be to amend the specification. Besides the fact that the specification says so I don't see any reason why this shouldn't be allowed.
'the specification says so' *IS* the reason. DT isn't a free for all where you get to do whatever you want, or whatever 'feels' right. It is supposed to be a stable, OS agnostic ABI. That means bindings have to follow the specs (when available).
Quoted 4 lines
Maybe a future revision will support PCI-E ECAM, but we don't know what that will look like, and I'm pretty sure you don't want to hold up your patches until an IEEE committee gets around to deciding something ;)
The Open Firmware Working Group quit doing things under the IEEE banner a long long time ago, precisely for the reason that the IEEE process was too slow, cumbersome, and expensive. The group thereafter published "Recommended Practices" documents under its own non-authority instead of going through the IEEE ratification process. The group's only "authority" arose from the fact that the members had expertise and worked for companies that needed to use the results.
The Working Group is now inactive. I am in contact with many of the former members, but by and large they are no longer interested in firmware standardization. Many of them work for companies that now hold their cards closely, that seemingly being the only way to make enough money to stay in business.
In other words, there is no committee.
New versions of the document in question, or of any document for that matter, can be issued by the simple expedient of interested parties coming to agreement and writing down that agreement. In practice, that is what happens in Linux land for nearly every issue.
At this time, the primary "interest group" for device tree issues revolves around the Linux kernel - the devicetree-discuss list is where all the action is. I suppose that Oracle still cares about Open Firmware on its SPARC systems, but it is likely that Oracle is going to do whatever they want no matter what Linux people say or do. In principle, the DT is supposed to be OS-agnostic, but in practice, which other OS both uses the DT and has enough users to matter? So it would be entirely reasonable for a group of Linux kernel people to publish revised documents, as has already been done in the form of Documentation/ files setting down usage rules.
Quoted 3 lines
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeQuoted 31 lines · Jason Gunthorpe
Quoted 4 lines · Thierry Reding
Quoted 2 lines
Not going down the of_pci_* code paths for address translation at the root port bridge nodes is certainly not right.
I'm not so sure. Why should the pcie-controller be a PCI device?
The spec is clear on that point as well:
device_type
A Standard Package conforming to this specification and corresponding to a device that implements a PCI bus shall implement this property with the string value "pci"The children of a pcie-controller node are PCI devices, thus the pcie-controller node 'implements a PCI bus'. Or, as you say 'device on the processor/SoC/platform bus that bridges to the PCI bus'
Think of
device_typeas also meaningbus_typeand it makes more logical sense, the name is terrible, but its usage is governed by the OF docs..Quoted 3 lines
The other alternative would be to amend the specification. Besides the fact that the specification says so I don't see any reason why this shouldn't be allowed.
'the specification says so' *IS* the reason. DT isn't a free for all where you get to do whatever you want, or whatever 'feels' right. It is supposed to be a stable, OS agnostic ABI. That means bindings have to follow the specs (when available).
Quoted 4 lines
Maybe a future revision will support PCI-E ECAM, but we don't know what that will look like, and I'm pretty sure you don't want to hold up your patches until an IEEE committee gets around to deciding something ;)
Mitch already answered this. The specification is now almost 15 years old and it couldn't possibly have foreseen all of the future use-cases. If the specification is too restrictive and Mitch gives his blessing to remove some of the restrictions, I don't see any reason why we shouldn't do so if it lets us represent the reality of hardware more accurately in DT.
Furthermore we're not discussing radical changes. None of the changes will be backwards-incompatible, but they will allow recent hardware to be represented more correctly or conveniently.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130313/901fb399/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 6 lines · Thierry Reding
Mitch already answered this. The specification is now almost 15 years old and it couldn't possibly have foreseen all of the future use-cases. If the specification is too restrictive and Mitch gives his blessing to remove some of the restrictions, I don't see any reason why we shouldn't do so if it lets us represent the reality of hardware more accurately in DT.
I understand the spec is old, and I have no problem with making a Linux specific revision, but do *that* - don't bury some random deviation inside the bindings for a driver. I even suggested some language, but I feel more thought is needed to consider how to model the standardized ECAM mechanism..
Quoted 3 lines
Furthermore we're not discussing radical changes. None of the changes will be backwards-incompatible, but they will allow recent hardware to be represented more correctly or conveniently.
Sure, but it is still inconsistent, one MMIO config mechansim is in ranges, one is in regs. Plus I don't think tegra's case is a great starting point to design a spec update, it's config access mechanism is especially strange...
Anyhow, I think this has been hashed to death, as long as your final binding has the 'device_type = pci' on the pcie-controller node I think it will be fine.
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Jason Gunthorpe
Quoted 6 lines · Thierry Reding
Mitch already answered this. The specification is now almost 15 years old and it couldn't possibly have foreseen all of the future use-cases. If the specification is too restrictive and Mitch gives his blessing to remove some of the restrictions, I don't see any reason why we shouldn't do so if it lets us represent the reality of hardware more accurately in DT.
Quoted 5 lines
I understand the spec is old, and I have no problem with making a Linux specific revision, but do *that* - don't bury some random deviation inside the bindings for a driver. I even suggested some language, but I feel more thought is needed to consider how to model the standardized ECAM mechanism..
As Mitch already said there's very little chance that the specification update will be ratified through IEEE, so I think that we might just as well put a corresponding text somewhere below Documentation/devicetree.
Also note that this has absolutely nothing to do with ECAM. All I'm proposing is to allow the reg property of a root port to be translated into a CPU addressable memory region through the ranges property. This turns out to actually work properly with the current OF core in Linux.
Quoted 8 lines
Quoted 3 lines
Furthermore we're not discussing radical changes. None of the changes will be backwards-incompatible, but they will allow recent hardware to be represented more correctly or conveniently.
Sure, but it is still inconsistent, one MMIO config mechansim is in ranges, one is in regs. Plus I don't think tegra's case is a great starting point to design a spec update, it's config access mechanism is especially strange...
Again, it is still the most accurate way to describe the hardware. I know it's not terribly beautiful and doesn't match with what you'd expect from PC hardware. However it is still a reality and something the kernel will have to deal with. Marvell hardware isn't very pretty either but that shouldn't exclude it from being supported by Linux.
Quoted 3 lines
Anyhow, I think this has been hashed to death, as long as your final binding has the '
device_type= pci' on the pcie-controller node I think it will be fine.
No. The pcie-controller is *not* a PCI device. It is a PCI host bridge. It is the instance that translates from the platform to the PCI bus. Why should it be device_type = "pci"? And we've also been over this before, making that change stops the proposed binding from working properly because the entry in the reg property can't be translated into a CPU addressable memory region.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130313/89f7970f/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 3 lines · Thierry Reding
As Mitch already said there's very little chance that the specification update will be ratified through IEEE, so I think that we might just as well put a corresponding text somewhere below
Documentation/devicetree.
Sure, I'm fine with that.
Quoted 4 lines
Also note that this has absolutely nothing to do with ECAM. All I'm proposing is to allow the reg property of a root port to be translated into a CPU addressable memory region through the ranges property. This turns out to actually work properly with the current OF core in Linux.
ECAM is the only standard based mechanism that could possibly use this new config space ranges concept. Whatever you do for tegra shouldn't stomp on the possibility of making that work in the future. So at least think about how ECAM could be modeled for a few mins before going ahead.
Tegra is the non-standard unusual case here, an update to the PCI OF binding should be of general use.
Quoted 7 lines
Quoted 3 lines
Anyhow, I think this has been hashed to death, as long as your final binding has the '
device_type= pci' on the pcie-controller node I think it will be fine.No. The pcie-controller is *not* a PCI device. It is a PCI host bridge. It is the instance that translates from the platform to the PCI bus. Why should it be
device_type= "pci"?
Spec says so, Mitch said so, other PCI host bridge bindings in the tree work like this. Do a grep in the powerpc tree, for instance.
arch/x86/platform/ce4100/falconfalls.dts is a pretty comprehensive example.
Quoted 3 lines
And we've also been over this before, making that change stops the proposed binding from working properly because the entry in the reg property can't be translated into a CPU addressable memory region.
Sigh, that's because the change makes things follow the OF PCI spec. The Linux OF core follows the spec. Config space ranges binding is specifically *defined to not work* in the spec. Why should it work today?
You have to deal with this problem somehow, and omitting the top level device_type is not a solution.
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 13 lines · Jason Gunthorpe
Quoted 3 lines · Thierry Reding
As Mitch already said there's very little chance that the specification update will be ratified through IEEE, so I think that we might just as well put a corresponding text somewhere below
Documentation/devicetree.Sure, I'm fine with that.
Quoted 4 lines
Also note that this has absolutely nothing to do with ECAM. All I'm proposing is to allow the reg property of a root port to be translated into a CPU addressable memory region through the ranges property. This turns out to actually work properly with the current OF core in Linux.
Quoted 8 lines
ECAM is the only standard based mechanism that could possibly use this new config space ranges concept. Whatever you do for tegra shouldn't stomp on the possibility of making that work in the future. So at least think about how ECAM could be modeled for a few mins before going ahead.
Tegra is the non-standard unusual case here, an update to the PCI OF binding should be of general use.
But I'm not trying to encode ECAM in the DT. ECAM isn't the problem at all. The only problem is that each root port has a separate window in the CPU address space that is used to access that port's configuration space.
ECAM only comes into play when accessing the configuration space of subordinate devices (via type 1 transactions). But at that point the device tree doesn't need to know about it. The only thing that needs to be in DT is the memory region that the controller should write to in order to generate type 1 transactions. And that's a problem that has long been solved.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Anyhow, I think this has been hashed to death, as long as your final binding has the '
device_type= pci' on the pcie-controller node I think it will be fine.No. The pcie-controller is *not* a PCI device. It is a PCI host bridge. It is the instance that translates from the platform to the PCI bus. Why should it be
device_type= "pci"?
Quoted 5 lines
Spec says so, Mitch said so, other PCI host bridge bindings in the tree work like this. Do a grep in the powerpc tree, for instance.
arch/x86/platform/ce4100/falconfalls.dtsis a pretty comprehensive example.
Yes, yes. We all know by now what the spec says. The whole point is that maybe the spec needs to be updated and this requirement lifted. What I keep trying to say is that in case of Tegra the PCIe controller does not provide the PCI bus. It doesn't appear as a device on the PCI bus and it doesn't have configuration space itself.
Only the PCI root ports provide access to the subordinate busses. And only they appear on the PCI bus.
Quoted 8 lines
Quoted 3 lines
And we've also been over this before, making that change stops the proposed binding from working properly because the entry in the reg property can't be translated into a CPU addressable memory region.
Sigh, that's because the change makes things follow the OF PCI spec. The Linux OF core follows the spec. Config space ranges binding is specifically *defined to not work* in the spec. Why should it work today?
Okay. I keep understanding what you're saying. But the whole point of this discussion is that maybe the spec should be amended to allow this to work. In fact it does work with the current implementation in the Linux kernel. And what I've been trying to explain is that I think this to be the most accurate description of the Tegra hardware.
Quoted 2 lines
You have to deal with this problem somehow, and omitting the top level
device_typeis not a solution.
So we've already agreed that the specification might be too restrictive, and I'm trying to solve this problem by finding an accurate description in DT to represent the hardware without following the spec to the letter because it wasn't written with this kind of hardware in mind.
Tegra's PCIe controller doesn't actually implement the bus. In fact we have to jump through hoops to make both root ports appear on the root bus specifically because there's no common access to the configuration space that includes the host bridge or the root ports.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130313/0239c411/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingQuoted 40 lines · Thierry Reding
Quoted 9 lines · Jason Gunthorpe
Quoted 6 lines · Thierry Reding
Mitch already answered this. The specification is now almost 15 years old and it couldn't possibly have foreseen all of the future use-cases. If the specification is too restrictive and Mitch gives his blessing to remove some of the restrictions, I don't see any reason why we shouldn't do so if it lets us represent the reality of hardware more accurately in DT.
Quoted 5 lines
I understand the spec is old, and I have no problem with making a Linux specific revision, but do *that* - don't bury some random deviation inside the bindings for a driver. I even suggested some language, but I feel more thought is needed to consider how to model the standardized ECAM mechanism..
As Mitch already said there's very little chance that the specification update will be ratified through IEEE, so I think that we might just as well put a corresponding text somewhere below
Documentation/devicetree.Also note that this has absolutely nothing to do with ECAM. All I'm proposing is to allow the reg property of a root port to be translated into a CPU addressable memory region through the ranges property. This turns out to actually work properly with the current OF core in Linux.
Quoted 8 lines
Quoted 3 lines
Furthermore we're not discussing radical changes. None of the changes will be backwards-incompatible, but they will allow recent hardware to be represented more correctly or conveniently.
Sure, but it is still inconsistent, one MMIO config mechansim is in ranges, one is in regs. Plus I don't think tegra's case is a great starting point to design a spec update, it's config access mechanism is especially strange...
Again, it is still the most accurate way to describe the hardware. I know it's not terribly beautiful and doesn't match with what you'd expect from PC hardware. However it is still a reality and something the kernel will have to deal with. Marvell hardware isn't very pretty either but that shouldn't exclude it from being supported by Linux.
Quoted 7 lines
Quoted 3 lines
Anyhow, I think this has been hashed to death, as long as your final binding has the '
device_type= pci' on the pcie-controller node I think it will be fine.No. The pcie-controller is *not* a PCI device. It is a PCI host bridge. It is the instance that translates from the platform to the PCI bus. Why should it be
device_type= "pci"?
device_type="pci" means that this device implements PCI functionality, which means that it is either a PCI host bridge or a PCI-to-PCI bridge.
"device_type" answers the question "What does this device do?". The alternative question "What bus does this device plug into?" is answered by looking at the parent.
In this case, the answer to "what does pcie_controller do?" is "it implements a PCI bus" below. So 'device_type = "pci"' is appropriate.
Having just re-read the code in drivers/of/address.c, I think I have a deeper understanding of a few practical issues:
In order to translate a 3/2 PCI address via of_get_pci_address(), the parent device (i.e. root complex / pcie-controller) must have both 'device_type = "pci"' (else of_match_bus() won't find the pci version of "struct of_bus") and also 'name = "pci"' (else strcmp(bus->name, "pci") will fail).
So it would seem that, if you want to use address.c verbatim (for the interface between pcie-controller and its direct children), not only do you need device_type = pci, but you also need to rename "pcie-controller" to "pci".
But then you run into the problem that the pci variant of struct of_bus uses "assigned-addresses" instead of "reg". So it still doesn't work as-is. To make it work, you would need to add an "assigned-addresses" property to each root port node. That property value could be in non-relocatable memory space, translatable via normal rules, in which case the "map config space to MMIO via ranges" discussion is moot for Marvell.
Which re-raises a question for which I have not yet heard a compelling
answer: Why use 3/2 PCI addressing between Marvell pcie-controller and its root ports (with their unusual programming model)? The Marvell hardware address structure between controller and root ports is definitely not 3/2. Certainly you need 3/2 addressing below (implemented by) the root ports, but I just don't see the utility of pretending 3/2 addressing at the complex-to-port interface. If the root port hardware used the common programming model, with real config headers and stuff, 3/2 would be good because you could use existing drivers. But since you need a special root-port driver anyway, why go to the trouble of emulating non-existent addressing?
And we've also been over this before,
Quoted 6 lines
making that change stops the proposed binding from working properly because the entry in the reg property can't be translated into a CPU addressable memory region.
Thierry
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]
Quoted 2 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.
Alright, that's 2 against 1. I don't have much of a choice but to yield.
Quoted 9 lines
Having just re-read the code in
drivers/of/address.c, I think I have a deeper understanding of a few practical issues:In order to translate a 3/2 PCI address via
of_get_pci_address(), the parent device (i.e. root complex / pcie-controller) must have both 'device_type= "pci"' (elseof_match_bus()won't find the pci version of "structof_bus") and also 'name = "pci"' (elsestrcmp(bus->name, "pci")will fail).
Quoted 4 lines
So it would seem that, if you want to use
address.cverbatim (for the interface between pcie-controller and its direct children), not only do you needdevice_type= pci, but you also need to rename "pcie-controller" to "pci".
bus->name refers to that of the of_bus structure, not the node name of the controller.
Quoted 7 lines
But then you run into the problem that the pci variant of struct
of_bususes "assigned-addresses" instead of "reg". So it still doesn't work as-is. To make it work, you would need to add an "assigned-addresses" property to each root port node. That property value could be in non-relocatable memory space, translatable via normal rules, in which case the "map config space to MMIO via ranges" discussion is moot for Marvell.
At the risk of quoting the spec: it says (4.1.2) that the entries in the "assigned-addresses" property correspond to the function's configuration space BARs. In this case however that's no longer true. But if everybody thinks that repurposing the "assigned-addresses" property is more acceptable than leaving out the device_type = "pci" property from the pcie-controller node, then so be it.
Quoted 11 lines
Which re-raises a question for which I have not yet heard a compelling
answer: Why use 3/2 PCI addressing between Marvell pcie-controller and its root ports (with their unusual programming model)? The Marvell hardware address structure between controller and root ports is definitely not 3/2. Certainly you need 3/2 addressing below (implemented by) the root ports, but I just don't see the utility of pretending 3/2 addressing at the complex-to-port interface. If the root port hardware used the common programming model, with real config headers and stuff, 3/2 would be good because you could use existing drivers. But since you need a special root-port driver anyway, why go to the trouble of emulating non-existent addressing?
I think the reason is that reg needs to be in the standard format for the device node matching code to work. There are various checks to make sure the reg property has entries which are 3/2 each.
There's also the issue that the pcie-controller's ranges property contains the mapping of the I/O and memory windows and therefore needs the 3/2 addressing to encode the type.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130313/8bd653a8/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 6 lines · Thierry Reding
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]Quoted 2 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.Alright, that's 2 against 1. I don't have much of a choice but to yield.
All issues of "voting" aside, 'device_type = pci' is what tells of_get_pci_address() to use the 3/2 interpretation. So if you want a node to implement 3/2 addresses, it must say device_type = pci, unless you do address translation some other way.
Quoted 10 lines
Quoted 10 lines
Having just re-read the code in
drivers/of/address.c, I think I have a deeper understanding of a few practical issues:In order to translate a 3/2 PCI address via
of_get_pci_address(), the parent device (i.e. root complex / pcie-controller) must have both 'device_type= "pci"' (elseof_match_bus()won't find the pci version of "structof_bus") and also 'name = "pci"' (elsestrcmp(bus->name, "pci")will fail).
Quoted 7 lines
Quoted 4 lines
So it would seem that, if you want to use
address.cverbatim (for the interface between pcie-controller and its direct children), not only do you needdevice_type= pci, but you also need to rename "pcie-controller" to "pci".
bus->namerefers to that of theof_busstructure, not the node name of the controller.
D'oh! Completely correct. That's good; I didn't like the idea of needing both device_type and name = pci.
Quoted 9 lines
Quoted 8 lines
But then you run into the problem that the pci variant of struct
of_bususes "assigned-addresses" instead of "reg". So it still doesn't work as-is. To make it work, you would need to add an "assigned-addresses" property to each root port node. That property value could be in non-relocatable memory space, translatable via normal rules, in which case the "map config space to MMIO via ranges" discussion is moot for Marvell.
Quoted 3 lines
At the risk of quoting the spec: it says (4.1.2) that the entries in the "assigned-addresses" property correspond to the function's configuration space BARs. In this case however that's no longer true.
Right, but we got into this mess by pretending that a PCI domain exists despite the fact that the hardware is not PCI at that level, in order to accommodate some existing Linux code. Since we are already "faking it to deal with existing code", it doesn't seem particularly strange to take advantage of how existing code works, in order to accomplish the forgery.
Quoted 4 lines
But if everybody
thinks that repurposing the "assigned-addresses" property is more acceptable than leaving out thedevice_type= "pci" property from the pcie-controller node, then so be it.
I'm confused again. Why does leaving out device_type = pci make it work?
Quoted 13 lines
Quoted 12 lines
Which re-raises a question for which I have not yet heard a compelling
answer: Why use 3/2 PCI addressing between Marvell pcie-controller and its root ports (with their unusual programming model)? The Marvell hardware address structure between controller and root ports is definitely not 3/2. Certainly you need 3/2 addressing below (implemented by) the root ports, but I just don't see the utility of pretending 3/2 addressing at the complex-to-port interface. If the root port hardware used the common programming model, with real config headers and stuff, 3/2 would be good because you could use existing drivers. But since you need a special root-port driver anyway, why go to the trouble of emulating non-existent addressing?
Quoted 3 lines
I think the reason is that reg needs to be in the standard format for the device node matching code to work. There are various checks to make sure the reg property has entries which are 3/2 each.
Could you point me to some of those checks so I can understand this better?
Quoted 4 lines
There's also the issue that the pcie-controller's ranges property contains the mapping of the I/O and memory windows and therefore needs the 3/2 addressing to encode the type.
Why can't that be done in each of the root port nodes?
Quoted 3 lines
Thierry
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 6 lines · Mitch Bradley
Quoted 5 lines · Thierry Reding
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]Quoted 2 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.
Quoted 6 lines
Quoted 1 line
Alright, that's 2 against 1. I don't have much of a choice but to yield.
All issues of "voting" aside, '
device_type= pci' is what tellsof_get_pci_address()to use the 3/2 interpretation. So if you want a node to implement 3/2 addresses, it must saydevice_type= pci, unless you do address translation some other way.
I should note that device_type is used for OF, but is supposed to not be used for FDT as matching against compatible properties is preferred. I don't have a good reason as to why, but Mitch may know the history. However, there are numerous exceptions to that for compatibility and to work with existing s/w. So this may be one of those cases.
Rob
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Rob Herring
Quoted 6 lines · Mitch Bradley
Quoted 5 lines · Thierry Reding
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]Quoted 2 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.Quoted 6 lines
Quoted 1 line
Alright, that's 2 against 1. I don't have much of a choice but to yield.
All issues of "voting" aside, '
device_type= pci' is what tellsof_get_pci_address()to use the 3/2 interpretation. So if you want a node to implement 3/2 addresses, it must saydevice_type= pci, unless you do address translation some other way.
Quoted 5 lines
I should note that
device_typeis used for OF, but is supposed to not be used for FDT as matching against compatible properties is preferred. I don't have a good reason as to why, but Mitch may know the history. However, there are numerous exceptions to that for compatibility and to work with existing s/w. So this may be one of those cases.
Indeed. The history is that, originally, the "name" property was supposed to be a very precise name, like what is now the most specific part of "compatible". And "device_type" was "what, in general, does this device do".
But with rule, pathnames quickly became unusable to humans:
/marvell,mv87230/marvell,svxyz/adaptec,12345/wd,8876
What the heck is that? Model names are like hashes; they have no rhyme or reason and humans cannot remember them for long. In order for a random person to have a clue about what something was, it was much better for pathname components to represent the generic function:
/pci/pci/scsi/disk
Now it's clear that we are looking at a bridged PCI domain hosting a SCSI disk. That's very useful for a system administrator; the gobbledygook name is not.
Furthermore, in light of the facts that model numbers change much more rapidly than programming interfaces, companies clone each others programming interfaces, and companies change their names, a much more capable way of describing programming models was needed.
So the "generic names" recommended practice changed things to make "name" mean what "device_type" used to vaguely mean, "device_type" was deprecated (it was never very precisely defined, and mutually-conflicting usage patterns had developed), and "compatible" in its present form was introduced. "Generic names" was quite controversial at the time - I think David Kahn wanted to strangle me at one point - but it was the right thing. (Not to mention the embarrassing mistake of the underscore in device_type, mea culpa.)
That being the case, I personally would like for device_type to disappear forever. But the fact exists that address.c currently uses it (and fdt.c duly parses it), which I why I said that, if you want to use of_get_pci_address() in its present form, device_type=pci must be present in the bus node that implements the 3/2 address domain.
So, yeah, it's compability with existing s/w, namely the PCI support in address.c .
Quoted 3 lines
Rob
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Mitch BradleyQuoted 6 lines · Mitch Bradley
Quoted 5 lines · Thierry Reding
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]Quoted 2 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.
Quoted 6 lines
Quoted 1 line
Alright, that's 2 against 1. I don't have much of a choice but to yield.
All issues of "voting" aside, '
device_type= pci' is what tellsof_get_pci_address()to use the 3/2 interpretation. So if you want a node to implement 3/2 addresses, it must saydevice_type= pci, unless you do address translation some other way.
of_bus_default_map() can also handle #address-cells > 2. We explicitly added that special case back when the bindings still looked very different and we needed to match the exact entry.
This is the reason why it actually works to not use device_type = "pci" for the pcie-controller. For subordinate device address translation the code will look at the parent bus, see that it isn't a PCI device and use the default bus, which in turn will simply memcmp() the first 3 cells of the "reg" property entry against those in the "ranges" property.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
But then you run into the problem that the pci variant of struct
of_bususes "assigned-addresses" instead of "reg". So it still doesn't work as-is. To make it work, you would need to add an "assigned-addresses" property to each root port node. That property value could be in non-relocatable memory space, translatable via normal rules, in which case the "map config space to MMIO via ranges" discussion is moot for Marvell.
Quoted 7 lines
Quoted 3 lines
At the risk of quoting the spec: it says (4.1.2) that the entries in the "assigned-addresses" property correspond to the function's configuration space BARs. In this case however that's no longer true.
Right, but we got into this mess by pretending that a PCI domain exists despite the fact that the hardware is not PCI at that level, in order to accommodate some existing Linux code.
Actually at that level the hardware is indeed PCI. These entries are for the root port nodes, which are in fact the bridges to the PCI bus. This is what I've been trying to say all along. The PCIe controller is just a collection of registers that allow CPU address space to be bridged to an internal bus (FPCI, which is in fact a relic from the HyperTransport roots of the controller IIUC) and setup/handle MSI.
The transition to the PCI fabric happens in the root ports. There is no PCI host bridge.
Quoted 3 lines
Since we are already "faking it to deal with existing code", it doesn't seem particularly strange to take advantage of how existing code works, in order to accomplish the forgery.
So we're back to square one. How is this kind of faking (repurposing the "assigned-addresses" property) any better than translating the "reg" property through an "ss = 00" entry in the "ranges" property?
Quoted 6 lines
Quoted 4 lines
But if everybody
thinks that repurposing the "assigned-addresses" property is more acceptable than leaving out thedevice_type= "pci" property from the pcie-controller node, then so be it.I'm confused again. Why does leaving out
device_type= pci make it work?
I've already said this above. The default bus implementation handles the case where #address-cells > 2 by comparing the first three cells, but not go through the additional range matching that would otherwise need to take place. Instead it treats the address cells as some kind of identifier to match between the "reg" and "ranges" entries. That's very much the way that the PCI bus implementation works, except for some extra PCI-specific checks.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Which re-raises a question for which I have not yet heard a compelling
answer: Why use 3/2 PCI addressing between Marvell pcie-controller and its root ports (with their unusual programming model)? The Marvell hardware address structure between controller and root ports is definitely not 3/2. Certainly you need 3/2 addressing below (implemented by) the root ports, but I just don't see the utility of pretending 3/2 addressing at the complex-to-port interface. If the root port hardware used the common programming model, with real config headers and stuff, 3/2 would be good because you could use existing drivers. But since you need a special root-port driver anyway, why go to the trouble of emulating non-existent addressing?
Quoted 5 lines
Quoted 3 lines
I think the reason is that reg needs to be in the standard format for the device node matching code to work. There are various checks to make sure the reg property has entries which are 3/2 each.
Could you point me to some of those checks so I can understand this better?
The code for that is in drivers/of/of_pci.c. The __of_pci_pci_compare() function looks like this:
static inline int __of_pci_pci_compare(struct device_node *node,
unsigned int devfn)
{
unsigned int size;
const __be32 *reg = of_get_property(node, "reg", &size);
if (!reg || size < 5 * sizeof(__be32))
return 0;
return ((be32_to_cpup(®[0]) >> 8) & 0xff) == devfn;
}This is the function that's called for each node to check whether it corresponds to a given PCI device by matching the devfn to the value found in the first cell of a device node's "reg" property.
Quoted 5 lines
Quoted 3 lines
There's also the issue that the pcie-controller's ranges property contains the mapping of the I/O and memory windows and therefore needs the 3/2 addressing to encode the type.
Why can't that be done in each of the root port nodes?
It could possibly be done, but we'd be lying. The way the hardware works is that we configure a single window to map to each of the FPCI regions for I/O and (non-)prefetchable memory spaces. So the pcie-controller node's "ranges" property contains three entries, one for each of those spaces which the driver uses to program the corresponding registers. The same information is passed to the resource assignment code of the PCI core so that it can partition these windows as appropriate and allocate sub-windows that are assigned to the BARs of the root-ports.
Note that the programming of the root-port BARs is vital here. If memory within one of the FPCI-mapped windows is accessed, the hardware will look for a matching address in the root port BARs in order to determine the port through which the transactions should be sent.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130314/52f7d3ec/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingQuoted 8 lines · Thierry Reding
On Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]Quoted 3 lines
In this case, the answer to "what does
pcie_controllerdo?" is "it implements a PCI bus" below. So 'device_type= "pci"' is appropriate.Alright, that's 2 against 1. I don't have much of a choice but to yield.
Just one note here though re: how the Tegra HW works:
The Tegra "PCIe controller" HW translates from a SoC-internal bus to another SoC-internal bus. The "PCIe root ports" translate from that second SoC-internal bus to a PCIe bus. That's exactly why the PCIe root port configuration registers don't show up via type 0 PCIe configuration transactions. Thus, from a HW perspective, it really is true that the PCIe root ports are PCI devices, but the "PCIe controller" really isn't anything to do with PCIe.
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Mitch BradleyOn Wed, Mar 13, 2013 at 10:58:02AM -1000, Mitch Bradley wrote:
[...]
Quoted 7 lines
But then you run into the problem that the pci variant of struct
of_bususes "assigned-addresses" instead of "reg". So it still doesn't work as-is. To make it work, you would need to add an "assigned-addresses" property to each root port node. That property value could be in non-relocatable memory space, translatable via normal rules, in which case the "map config space to MMIO via ranges" discussion is moot for Marvell.
So if I understand all of the above correctly, the Tegra binding should look like this:
pcie-controller {
compatible = "nvidia,tegra20-pcie";
device_type = "pci";
reg = <0x80003000 0x00000800 /* PADS registers */
0x80003800 0x00000200 /* AFI registers */
0x90000000 0x10000000>; /* configuration space */
reg-names = "pads", "afi", "cs";
interrupts = <0 98 0x04 /* controller interrupt */
0 99 0x04>; /* MSI interrupt */
interrupt-names = "intr", "msi";
bus-range = <0x00 0xff>;
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x82000800 0 0 0x80000000 0 0x00001000 /* port 0 registers */
0x82001000 0 0 0x80001000 0 0x00001000 /* port 1 registers */
0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */
0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
clocks = <&tegra_car 70>, <&tegra_car 72>, <&tegra_car 74>,
<&tegra_car 118>;
clock-names = "pex", "afi", "pcie_xclk", "pll_e";
status = "disabled";
pci at 1,0 {
device_type = "pci";
assigned-addresses = <0x82000800 0 0 0x80000000 0x1000>;
reg = <0x000800 0 0 0 0>;
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
nvidia,num-lanes = <2>;
};
pci at 2,0 {
device_type = "pci";
assigned-addresses = <0x82001000 0 0 0x80001000 0x1000>;
reg = <0x001000 0 0 0 0>;
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
nvidia,num-lanes = <2>;
};
};Does that look about correct?
One problem I see is that now the ranges property contains three entries for 32-bit memory space:
0x82000800 0 0 0x80000000 0 0x00001000 /* port 0 registers */ 0x82001000 0 0 0x80001000 0 0x00001000 /* port 1 registers */ 0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */
But we used to rely on the ss field to parse the ranges property and configure the windows in the controller properly so that accesses to these regions will generate the correct transactions.
Now the code will actually match the first entry and assume that it represents the non-prefetchable 32-bit memory space and program the controller to forward accesses to the 0x80000000-0x80000fff region as PCI memory write transactions.
I suppose we could add a check to verify that devfn == 0. However does the above ranges property not imply that the brige on bus 0, device 1 has a 32-bit memory space from 0x80000000 to 0x80000fff and the bridge on bus 0, device 2 has a 32-bit memory space from 0x80001000 to 0x80001fff? Which matches with the wording in the spec that the "assigned-addresses" correspond to the bridge's BARs.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130313/0a8d2cfa/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Thierry Reding
Does that look about correct?
By my reading of the spec the entries in ranges should not have the b,d,f bits set..
Although it is not a big stretch to include them..
Quoted 4 lines
Now the code will actually match the first entry and assume that it represents the non-prefetchable 32-bit memory space and program the controller to forward accesses to the 0x80000000-0x80000fff region as PCI memory write transactions.
Yah, it makes a mess of determining the host bridge aperture. Require the aperture to be first/last?
Make use of the 2nd dword?
Go back to the idea of the top level reg?
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Jason Gunthorpe
Quoted 1 line · Thierry Reding
Does that look about correct?
By my reading of the spec the entries in ranges should not have the b,d,f bits set..
Although it is not a big stretch to include them..
Quoted 4 lines
Now the code will actually match the first entry and assume that it represents the non-prefetchable 32-bit memory space and program the controller to forward accesses to the 0x80000000-0x80000fff region as PCI memory write transactions.
Quoted 6 lines
Yah, it makes a mess of determining the host bridge aperture. Require the aperture to be first/last?
Make use of the 2nd dword?
Go back to the idea of the top level reg?
It turns out that this works with the Tegra driver because it uses the new of_pci_process_ranges() function and simply overwrites earlier matches by subsequent ones.
ranges = <0x82000000 0 0 0x80000000 0 0x00001000 /* port 0 registers */ 0x82000000 0 0 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
So the second entry would overwrite the non-prefetchable memory region as established by the first entry. Finally that will also be overwritten by the fourth entry.
So the above will actually work along with the corresponding root-port "assigned-addresses" properties. I still don't like it much because I don't think it accurately reflects the hardware. Furthermore it has the same kludgy, non-spec conformant smack that my original proposal had because it uses assigned-addresses for something it wasn't intended to. Also the mapping is actually not correct, because the first two entries don't refer to 32-bit memory space. They really map configuration space and therefore the ss field is wrong IMO (it should be 00, but that won't work with the OF core). One of the very early versions of my Tegra PCIe patch series actually introduced an IORESOURCE_CS type to account for this but it didn't go anywhere and became obsolete when I switched to the new binding which side-steps this issue by not using the PCI bus' translation mechanism.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130314/dd760e71/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
[trimm'd the cc list]
Quoted 9 lines · Thierry Reding
It turns out that this works with the Tegra driver because it uses the new
of_pci_process_ranges()function and simply overwrites earlier matches by subsequent ones.ranges = <0x82000000 0 0 0x80000000 0 0x00001000 /* port 0 registers */ 0x82000000 0 0 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
Okay.. There is still something funny here, the 3rd dword of the child address should not be 0 in every line and there shouldn't be overlaps in the child address space.
I'm assuming 0x80000000, 0xa0000000 and 0xb0000000 are real CPU physical addresses?
Then it should probably look like:
ranges = <0x82000000 0 0x80000000 0x80000000 0 0x00001000 /* port 0 registers */
0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */
Which says 'access to CPU address 0xa0000000 produces a PCI-E memory TLP with address 0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?
It also says 'access to CPU address 0x82000000 produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..
Then you'd go on to have:
pci at 1,0 {
device_type = "pci";
assigned-addresses = <0x82000000 0 0x80000000 0 0x1000>;
reg = <0x000800 0 0 0 0>;
}
pci at 2,0 {
device_type = "pci";
assigned-addresses = <0x82000000 0 0x80001000 0 0x1000>;
reg = <0x001000 0 0 0 0>;
}Notice I've made the upper dw of assigned-addresses's size 0 and included the full 3dw from the appropriate ranges line.
Quoted 3 lines
So the above will actually work along with the corresponding root-port "assigned-addresses" properties. I still don't like it much because I don't think it accurately reflects the hardware.
There are lots of valid ways to model the same HW :(
Bear in mind, for the PCI case - the OF PCI bindings model the HW through the eyes of the abstractions in the PCI specification. That is to say, they are not supposed to be an exact representation of the on chip architecture.
Perhaps this would be clearer if you used 'pcie-root-complex' as the name of the top level node?
Quoted 3 lines
same kludgy, non-spec conformant smack that my original proposal had because it uses assigned-addresses for something it wasn't intended to.
Yes, only the top level 'reg' method avoids going outside any specs.
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Jason Gunthorpe
[trimm'd the cc list]
Quoted 9 lines · Thierry Reding
It turns out that this works with the Tegra driver because it uses the new
of_pci_process_ranges()function and simply overwrites earlier matches by subsequent ones.ranges = <0x82000000 0 0 0x80000000 0 0x00001000 /* port 0 registers */ 0x82000000 0 0 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */
Quoted 6 lines
Okay.. There is still something funny here, the 3rd dword of the child address should not be 0 in every line and there shouldn't be overlaps in the child address space.
I'm assuming
0x80000000,0xa0000000and0xb0000000are real CPU physical addresses?
Yes.
Quoted 29 lines
Then it should probably look like:
ranges = <
0x8200000000x800000000x8000000000x00001000/* port 0 registers */0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */Which says 'access to CPU address
0xa0000000produces a PCI-E memory TLP with address0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?It also says 'access to CPU address
0x82000000produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..Then you'd go on to have:
pci at 1,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80000000 0 0x1000>; reg = <0x000800 0 0 0 0>; } pci at 2,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80001000 0 0x1000>; reg = <0x001000 0 0 0 0>; }
Quoted 2 lines
Notice I've made the upper dw of assigned-addresses's size 0 and included the full 3dw from the appropriate ranges line.
Okay, that all makes sense.
Quoted 10 lines
Quoted 3 lines
So the above will actually work along with the corresponding root-port "assigned-addresses" properties. I still don't like it much because I don't think it accurately reflects the hardware.
There are lots of valid ways to model the same HW :(
Bear in mind, for the PCI case - the OF PCI bindings model the HW through the eyes of the abstractions in the PCI specification. That is to say, they are not supposed to be an exact representation of the on chip architecture.
That seems to be at odds with most other uses of DT that I've come across. Generally the guideline seems to be to describe hardware irrespective of the underlying implementation and leave it up to the driver to translate the DT description into something the OS can use.
Quoted 2 lines
Perhaps this would be clearer if you used 'pcie-root-complex' as the name of the top level node?
Perhaps. I'm not sure.
Quoted 5 lines
Quoted 3 lines
same kludgy, non-spec conformant smack that my original proposal had because it uses assigned-addresses for something it wasn't intended to.
Yes, only the top level 'reg' method avoids going outside any specs.
Yes. It has a couple of other disadvantages, though, so if what we've been discussing here is in any way acceptable I'd rather go with that solution, even if I'm not entirely happy about it either.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130314/b7e57fbb/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 3 lines · Thierry Reding
Quoted 3 lines
pci at 1,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80000000 0 0x1000>;
Sorry, I missed this. The b,d,f bits should be set:
assigned-addresses = <0x82000800 0 0x80000000 0 0x1000>;
Quoted 5 lines
Quoted 5 lines
reg = <0x000800 0 0 0 0>; } pci at 2,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80001000 0 0x1000>;
assigned-addresses = <0x82001000 0 0x80001000 0 0x1000>;
The 'ranges' stays the same, the PCI 3/2 parse will ignore the b,d,f bits when comparing against ranges.
The use of r == 0 for assigned-addresses should be enough to disambiguate this case from any future use of assigned-addresses - the normal case has r == offset of the associated BAR != 0.
Cheers,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingQuoted 69 lines · Thierry Reding
Quoted 14 lines · Jason Gunthorpe
[trimm'd the cc list]
Quoted 9 lines · Thierry Reding
It turns out that this works with the Tegra driver because it uses the new
of_pci_process_ranges()function and simply overwrites earlier matches by subsequent ones.ranges = <0x82000000 0 0 0x80000000 0 0x00001000 /* port 0 registers */ 0x82000000 0 0 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0 0xb0000000 0 0x10000000>; /* prefetchable memory */Quoted 6 lines
Okay.. There is still something funny here, the 3rd dword of the child address should not be 0 in every line and there shouldn't be overlaps in the child address space.
I'm assuming
0x80000000,0xa0000000and0xb0000000are real CPU physical addresses?Yes.
Quoted 29 lines
Then it should probably look like:
ranges = <
0x8200000000x800000000x8000000000x00001000/* port 0 registers */0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */Which says 'access to CPU address
0xa0000000produces a PCI-E memory TLP with address0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?It also says 'access to CPU address
0x82000000produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..Then you'd go on to have:
pci at 1,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80000000 0 0x1000>; reg = <0x000800 0 0 0 0>; } pci at 2,0 { device_type = "pci"; assigned-addresses = <0x82000000 0 0x80001000 0 0x1000>; reg = <0x001000 0 0 0 0>; }Quoted 2 lines
Notice I've made the upper dw of assigned-addresses's size 0 and included the full 3dw from the appropriate ranges line.
Okay, that all makes sense.
Quoted 10 lines
Quoted 3 lines
So the above will actually work along with the corresponding root-port "assigned-addresses" properties. I still don't like it much because I don't think it accurately reflects the hardware.
There are lots of valid ways to model the same HW :(
Bear in mind, for the PCI case - the OF PCI bindings model the HW through the eyes of the abstractions in the PCI specification. That is to say, they are not supposed to be an exact representation of the on chip architecture.
Quoted 4 lines
That seems to be at odds with most other uses of DT that I've come across. Generally the guideline seems to be to describe hardware irrespective of the underlying implementation and leave it up to the driver to translate the DT description into something the OS can use.
This can be a gray area where one just has to make a judgement call. You can think of the hardware at various levels of abstraction and sometimes it is not obvious where to draw the line. "Tell the truth" tends to be more stable in the long run, but can be more work in the short run, and it can be hard to predict the future well enough to choose the optimum approach.
Quoted 18 lines
Quoted 3 lines
Perhaps this would be clearer if you used 'pcie-root-complex' as the name of the top level node?
Perhaps. I'm not sure.
Quoted 5 lines
Quoted 3 lines
same kludgy, non-spec conformant smack that my original proposal had because it uses assigned-addresses for something it wasn't intended to.
Yes, only the top level 'reg' method avoids going outside any specs.
Yes. It has a couple of other disadvantages, though, so if what we've been discussing here is in any way acceptable I'd rather go with that solution, even if I'm not entirely happy about it either.
Thierry
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeOn Thu, Mar 14, 2013 at 11:25:55AM -0600, Jason Gunthorpe wrote:
[...]
Quoted 11 lines
I'm assuming
0x80000000,0xa0000000and0xb0000000are real CPU physical addresses?Then it should probably look like:
ranges = <
0x8200000000x800000000x8000000000x00001000/* port 0 registers */0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */
Quoted 7 lines
Which says 'access to CPU address
0xa0000000produces a PCI-E memory TLP with address0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?It also says 'access to CPU address
0x82000000produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..
Both of the above paragraphs are true. However accesses to the windows at 0x80000000 and 0x80001000 don't generate memory TLPs but type 0 configuration space TLPs.
So my first instinct was to make the first cell of the first two entries 0, but that doesn't work, since the OF core expects to find either memory or I/O spaces. ss == 2 isn't quite right here.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130314/43cd68c5/attachment.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Thierry Reding
Quoted 10 lines
ranges = <
0x8200000000x800000000x8000000000x00001000/* port 0 registers */0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */Which says 'access to CPU address
0xa0000000produces a PCI-E memory TLP with address0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?
Quoted 7 lines
Quoted 3 lines
It also says 'access to CPU address
0x82000000produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..Both of the above paragraphs are true. However accesses to the windows at
0x80000000and0x80001000don't generate memory TLPs but type 0 configuration space TLPs.
By my understanding access to 0x80000000/0x80001000 doesn't generate any externally visible TLPs?
IHMO modeling this register space as a controller-internal MMIO region associated with the bridge is reasonable... After all, you are iomapping it and accessing it with readl/writel - those are MMIO functions..
Quoted 3 lines
So my first instinct was to make the first cell of the first two entries 0, but that doesn't work, since the OF core expects to find either memory or I/O spaces.
Right, there is no specified way to translate config ss through ranges.
Regards,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 16 lines · Jason Gunthorpe
Quoted 14 lines · Thierry Reding
Quoted 13 lines
ranges = <
0x8200000000x800000000x8000000000x00001000/* port 0 registers */0x82000000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ 0x81000000 0 0 0x82000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0xa0000000 0xa0000000 0 0x10000000 /* non-prefetchable memory */ 0xc2000000 0 0xb0000000 0xb0000000 0 0x10000000>; /* prefetchable memory */Which says 'access to CPU address
0xa0000000produces a PCI-E memory TLP with address0xa0000000' - this is the 'normal' case, I assume that is what happens on tegra?It also says 'access to CPU address
0x82000000produces a PCI-E IO TLP with address 0' - this translation is something Linux typically expects..
Quoted 6 lines
Quoted 3 lines
Both of the above paragraphs are true. However accesses to the windows at
0x80000000and0x80001000don't generate memory TLPs but type 0 configuration space TLPs.By my understanding access to 0x80000000/0x80001000 doesn't generate any externally visible TLPs?
Now that you mention it, that's probably correct, yes.
Quoted 4 lines
IHMO modeling this register space as a controller-internal MMIO region associated with the bridge is reasonable... After all, you are iomapping it and accessing it with readl/writel - those are MMIO functions..
Yes, I think that'd be okay.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130314/6ed5fe50/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Mitch BradleyQuoted 4 lines · Mitch Bradley
port hardware used the common programming model, with real config headers and stuff, 3/2 would be good because you could use existing drivers. But since you need a special root-port driver anyway, why go to the trouble of emulating non-existent addressing?
Yes, I'm sorry, I've tried not to explore the complexity behind this reasoning here.. Those discussions went on for a long time and I'm tired of rehashing them :)
It isn't existing drivers that are valuable, it is the entire common PCI infrastructure for dealing with discovery and address assignment that is has value.
The common infrastructure can only allocate addresses within fixed regions assigned to each PCI domain - and it divides those addreses regions between ports in the root complex via the standard PCI-E root port bridge memory windows.
Now, the largest SOC has 10 of the PCI-E ports in it, with 32 bit addressing. It is essential that a small region of address space be set aside for PCI-E use, and thus it is critical that resource assignment be done across all ports, drawing on a common pool of address space.
Thus two choices:
1) Keep the PCI core the same and let the host bridge driver provide the missing configuration elements of the root complex
2) Upgrade the PCI common core to deal with dynamic cross domain resource allocation, including hot plug.
Based on discussion/analysis #1 was chosen. Lots of reasons.
Quoted 1 line
From that choice flows the DT bindings, which are now required to
model the PCI bus 0.
Does that help explain the rational?
Cheers,
Jason
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingDear Thierry Reding,
Quoted 2 lines · Thierry Reding
Oh well. I don't like it, but if that's the way it has to be, then so be it.
I don't like it too much either, but it's not a big problem either. It works, and satisfies the DT requirements reasonably.
Quoted 4 lines
Any reason why the reg-names can't match the root port's node name so that it can be used directly instead of going through hoops to construct a string from extra parameters? Like below:
Quoted 12 lines
pcie-controller {
regs = <...>;
reg-names = ..., "pcie at 1,0", "pcie at 2,0";
pcie at 1,0 {
...
};
pcie at 2,0 {
...
};
};Yes, I thought about doing this, but in my driver, I also use the "pcie0.0" string to request the address decoding window from the mvebu-mbus driver. So I would anyway have to construct this "pciX.Y" string. But if you feel like using "pcie at X,Y" for the reg-names is better even if I still need to construct the "pcieX.Y" string, then I will be perfectly ok with making this change.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingQuoted 9 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 6 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
Quoted 8 lines
Quoted 2 lines
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
The compatible property of the PCI host controller can imply what each index of the reg property entries is for.
Quoted 5 lines
Adding a property in the root port nodes seems like a cleaner and more accurate representation of the hardware to me, but if that's not acceptable perhaps we need to bite the bullet and add the code to look the registers up from the parent's reg property.
What I don't like is a new property defined to describe mmio addresses. We already have a property for that and it is "reg". But I think I'm still missing something:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
pci at 0,1 { ... reg = <0x00000800 0 0 0 0>;
Is this a PCI bus address?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
regs = <0x80000000 0x00001000>; };
Rob
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 23 lines · Rob Herring
Quoted 9 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 6 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
Quoted 8 lines
Quoted 2 lines
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
The compatible property of the PCI host controller can imply what each index of the reg property entries is for.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Adding a property in the root port nodes seems like a cleaner and more accurate representation of the hardware to me, but if that's not acceptable perhaps we need to bite the bullet and add the code to look the registers up from the parent's reg property.
What I don't like is a new property defined to describe mmio addresses. We already have a property for that and it is "reg".
Okay, understood.
Quoted 7 lines
But I think I'm still missing something:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
pci at 0,1 { ... reg = <0x00000800 0 0 0 0>;Is this a PCI bus address?
Yes. The example is slightly wrong. pci at 0,1 should actually be pci at 1,0. That means it is device 1, function 0 on the bus (implicitly 0 in this case). That matches the values in the reg property, whose first cell is defined as follows, quoting the PCI OF specification:
npt000ss bbbbbbbb dddddfff rrrrrrrr
Where:
n is 0 if the address is relocatable, 1 otherwise
p is 1 if the addressable region is "prefetchable", 0
otherwise
t is 1 if the address is aliased (for non-relocatable
I/O), below 1 MB (for Memory), or below 64 KB (for relocatable I/O).
ss is the space code, denoting the address space
bbbbbbbb is the 8-bit Bus Number
ddddd is the 5-bit Device Number
fff is the 3-bit Function Number
rrrrrrrr is the 8-bit Register Number
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130309/5685c82c/attachment-0001.sig>
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Rob HerringQuoted 23 lines · Rob Herring
Quoted 9 lines · Thierry Reding
Quoted 1 line · Rob Herring
On 03/07/2013 02:47 PM, Thierry Reding wrote:
[...]
Quoted 6 lines
Quoted 5 lines
In a nutshell (since some of the context isn't quoted anymore) the problem that we're trying to solve is that some of the embedded SoCs require per-root-port registers for configuration. The PCI DT specification doesn't make any provisions for this. A few alternatives have been discussed so far:
Quoted 8 lines
Quoted 2 lines
I'm not sure I follow. This is different than the host controller registers? Why would this not just be multiple entries in the reg property?
Well the register regions are per root-port. On Tegra20 there's 2 of them, Tegra30 has 3 and if I understand correctly Marvell can have up to 10 (!). Adding all of them to the reg property of the host controller could work but it needs some way to match the reg entry to the root port similar to option 1 below.
The compatible property of the PCI host controller can imply what each index of the reg property entries is for.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Adding a property in the root port nodes seems like a cleaner and more accurate representation of the hardware to me, but if that's not acceptable perhaps we need to bite the bullet and add the code to look the registers up from the parent's reg property.
What I don't like is a new property defined to describe mmio addresses. We already have a property for that and it is "reg". But I think I'm still missing something:
As stated in my recent reply to Jason, I thing the correct property is "ranges". "Ranges" translates mappable child address space addresses into parent addresses, and that is exactly what is going on. A specific subset of config addresses is mappable into parent MMIO space.
When done that way, there is no need for the multi-entry reg property at the top level, and the correspondence between specific root port DT nodes and their MMIO addresses is easily determined by matching child reg properties to parent ranges entries.
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pci at 0,1 { ... reg = <0x00000800 0 0 0 0>;Is this a PCI bus address?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
regs = <0x80000000 0x00001000>; };Rob
_______________________________________________
devicetree-discuss mailing list
devicetree-discuss at lists.ozlabs.org
https://lists.ozlabs.org/listinfo/devicetree-discuss
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Mitch Bradley,
Quoted 4 lines · Mitch Bradley
As stated in my recent reply to Jason, I thing the correct property is "ranges". "Ranges" translates mappable child address space addresses into parent addresses, and that is exactly what is going on. A specific subset of config addresses is mappable into parent MMIO space.
The PCI configuration space is *not* mapped in the MMIO space on Marvell hardware. In the MMIO space of each PCIe interface, there are many registers, only *two* of which are dedicated to accessing the PCI configuration space:
* One register to set the offset in the PCI configuration space.
* One register to read or write a value in the PCI configuration, at the offset specified in the first register.
See the implementation of mvebu_pcie_hw_rd_conf() and mvebu_pcie_hw_wr_conf() in the driver.
So really, the values specified in the reg = <...> property are *not* the PCI configuration spaces mapped in the MMIO space. They represent a bunch of per PCIe interface registers used to configure them, get the status of the link... and access, through an indirect mechanism, the PCI configuration space.
Does this helps?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 22 lines · Thomas Petazzoni
Dear Mitch Bradley,
Quoted 4 lines · Mitch Bradley
As stated in my recent reply to Jason, I thing the correct property is "ranges". "Ranges" translates mappable child address space addresses into parent addresses, and that is exactly what is going on. A specific subset of config addresses is mappable into parent MMIO space.
The PCI configuration space is *not* mapped in the MMIO space on Marvell hardware. In the MMIO space of each PCIe interface, there are many registers, only *two* of which are dedicated to accessing the PCI configuration space:
* One register to set the offset in the PCI configuration space.
* One register to read or write a value in the PCI configuration, at the offset specified in the first register.
See the implementation of
mvebu_pcie_hw_rd_conf()andmvebu_pcie_hw_wr_conf()in the driver.
Quoted 7 lines
So really, the values specified in the reg = <...> property are *not* the PCI configuration spaces mapped in the MMIO space. They represent a bunch of per PCIe interface registers used to configure them, get the status of the link... and access, through an indirect mechanism, the PCI configuration space.
Does this helps?
I agree that PCI config space accesses to *downstream* devices is via an indirect-access register pair.
The question is, does that indirect-access mechanism apply also to the internal config headers for the root port bridges?
According to section 20.15 of the MV78230 functional spec that I am looking at, the configuration header registers are mapped to the internal memory space. That section is unclear about whether those registers are CPU-accessible via indirect-access config transactions. When the PCIe hardware is configured for endpoint mode, the internal headers can be accessed via PCIe config transactions from an external port, but in root complex mode, the possibility of indirect access from the CPU is not mentioned.
The manual is a little vague in some respects, but it does say quite clearly that MMIO access to the root port bridge config header is possible, and it lists the MMIO addresses thereof (section A.10).
So we all agree that access to external (downstream) config registers is via the indirect register pair. The unclear thing is whether the internal config registers for the root port bridges can be indirectly accessed.
Do we have empirical evidence that indirect-access works to the internal root port bridge config headers?
Quoted 5 lines
Thanks,
Thomas
Bjorn Helgaas
maintainerPATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas Petazzonidrivers/pci/host/Kconfig line 8
Quoted 11 lines · Thomas Petazzoni
drivers/pci/host/Kconfig +5
-0@@ -1,4 +1,10 @@ menu "PCI host controller drivers" depends on PCI +config PCI_MVEBU + bool "Marvell EBU PCIe controller" + depends on ARCH_MVEBU + select PCI_SW_HOST_BRIDGE + select PCI_SW_PCI_PCI_BRIDGE
I think PCI_SW_HOST_BRIDGE and PCI_SW_PCI_PCI_BRIDGE are obsolete and can be removed, right?
Quoted 121 lines
drivers/pci/host/pci-mvebu.c new file +117
-0+static void mvebu_pcie_setup_io_window (struct mvebu_pcie_port *port, +int enable) +{ +unsigned long iobase, iolimit; + +if (port->bridge.iolimit < port->bridge.iobase) +return ; + + iolimit =0xFFF | ((port->bridge.iolimit &0xF0 ) <<8 ) | + (port->bridge.iolimitupper <<16 ); + iobase = ((port->bridge.iobase &0xF0 ) <<8 ) | + (port->bridge.iobaseupper <<16 ); + +if (enable) { +unsigned long physbase = port->pcie->io.start + iobase; +armada_370_xp_alloc_pcie_window (port->port, port->lane, + physbase, iobase, + iolimit-iobase, +IORESOURCE_IO ); +pci_ioremap_io (iobase, physbase); + } +else +armada_370_xp_free_pcie_window (iobase); +} + +static void mvebu_pcie_setup_mem_window (struct mvebu_pcie_port *port, +int enable) +{ +unsigned long membase, memlimit; + +if (port->bridge.memlimit < port->bridge.membase) +return ; + + membase = ((port->bridge.membase &0xFFF0 ) <<16 ); + memlimit = ((port->bridge.memlimit &0xFFF0 ) <<16 ) |0xFFFFF ; + +if (enable) +armada_370_xp_alloc_pcie_window (port->port, port->lane, + membase,ORION_ADDR_MAP_NO_REMAP , + memlimit-membase, +IORESOURCE_MEM ); +else +armada_370_xp_free_pcie_window (membase); +} + +static void mvebu_handle_pcie_command (struct mvebu_pcie_port *port,u16 old, +u16 new) +{ +/* Enabling an I/O window ? */ +if (!(old &PCI_COMMAND_IO ) && (new &PCI_COMMAND_IO )) +mvebu_pcie_setup_io_window (port,1 ); + +/* Disabling an I/O window ? */ +if ((old &PCI_COMMAND_IO ) && !(new &PCI_COMMAND_IO )) +mvebu_pcie_setup_io_window (port,0 ); + +/* Enabling a memory window ? */ +if (!(old &PCI_COMMAND_MEMORY ) && (new &PCI_COMMAND_MEMORY )) +mvebu_pcie_setup_mem_window (port,1 ); + +/* Disabling a memory window ? */ +if ((old &PCI_COMMAND_MEMORY ) && !(new &PCI_COMMAND_MEMORY )) +mvebu_pcie_setup_mem_window (port,0 ); +} +/* Write to the PCI-to-PCI bridge configuration space */ +static int mvebu_sw_pci_bridge_write (struct mvebu_pcie_port *port, +unsigned int where,int size,u32 value) +{ +struct mvebu_sw_pci_bridge *bridge = &port->bridge; +u32 mask, reg; +int err; + +if (size ==4 ) + mask =0x0 ; +else if (size ==2 ) + mask = ~(0xffff << ((where &3 ) *8 )); +else if (size ==1 ) + mask = ~(0xff << ((where &3 ) *8 )); +else +return PCIBIOS_BAD_REGISTER_NUMBER ; + + err =mvebu_sw_pci_bridge_read (port, where & ~3 ,4 , ®); +if (err) +return err; + + value = (reg & mask) | value << ((where &3 ) *8 ); + +switch (where & ~3 ) { +case PCI_COMMAND : +mvebu_handle_pcie_command (port, bridge->command, + value &0xffff ); + bridge->command = value &0xffff ; + bridge->status = value >>16 ; +break ; + + case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1: + bridge->bar[((where & ~3 ) -PCI_BASE_ADDRESS_0 ) /4 ] = value; +break ; + +case PCI_IO_BASE : +/* +* We also keep bit 1 set, it is a read-only bit that +* indicates we support 32 bits addressing for the +* I/O +*/ + bridge->iobase = (value &0xff ) |PCI_IO_RANGE_TYPE_32 ; + bridge->iolimit = ((value >>8 ) &0xff ) |PCI_IO_RANGE_TYPE_32 ; + bridge->secondary_status = value >>16 ; +break ; + +case PCI_MEMORY_BASE : + bridge->membase = value &0xffff ; + bridge->memlimit = value >>16 ; +break ; + +case PCI_PREF_MEMORY_BASE : + bridge->prefmembase = value &0xffff ;
drivers/pci/host/pci-mvebu.c line 241
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ bridge->prefmemlimit = value >>16 ; +break ; + +case PCI_PREF_BASE_UPPER32 : + bridge->prefbaseupper = value; +break ; + +case PCI_PREF_LIMIT_UPPER32 : + bridge->preflimitupper = value; +break ;
You're relying on a subsequent PCI_COMMAND write to set PCI_COMMAND_IO and/or PCI_COMMAND_MEMORY, and you program the bridge windows at that time. It might be a good idea if the PCI core did clear those bits while updating the windows, but I'm not sure we do. In any case, delaying the update is a difference from a standard P2P bridge that could cause issues later.
Quoted 9 lines
drivers/pci/host/pci-mvebu.c new file +9
-0+case PCI_IO_BASE_UPPER16 : + bridge->iobaseupper = value &0xffff ; + bridge->iolimitupper = value >>16 ; +break ; + +case PCI_PRIMARY_BUS : + bridge->primary_bus = value &0xff ; + bridge->secondary_bus = (value >>8 ) &0xff ; + bridge->subordinate_bus = (value >>16 ) &0xff ;
drivers/pci/host/pci-mvebu.c line 346
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ bridge->secondary_latency_timer = (value >>24 ) &0xff ; +orion_pcie_set_local_bus_nr (port->base, bridge->secondary_bus); +break ; + +default : +break ; + } + +return PCIBIOS_SUCCESSFUL ; +}
Bjorn
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Bjorn Helgaas,
Quoted 8 lines · Bjorn Helgaas
drivers/pci/host/Kconfig line 8
Quoted 5 lines
drivers/pci/host/Kconfig +5
-0+config PCI_MVEBU + bool "Marvell EBU PCIe controller" + depends on ARCH_MVEBU + select PCI_SW_HOST_BRIDGE + select PCI_SW_PCI_PCI_BRIDGEI think
PCI_SW_HOST_BRIDGEandPCI_SW_PCI_PCI_BRIDGEare obsolete and can be removed, right?
Indeed, will fix.
Quoted 7 lines
drivers/pci/host/pci-mvebu.c line 326
Quoted 3 lines
drivers/pci/host/pci-mvebu.c new file +3
-0+case PCI_PREF_LIMIT_UPPER32 : + bridge->preflimitupper = value; +break ;You're relying on a subsequent
PCI_COMMANDwrite to setPCI_COMMAND_IOand/orPCI_COMMAND_MEMORY, and you program the bridge windows at that time.
Correct.
Quoted 4 lines
It might be a good idea if the PCI core did clear those bits
while updating the windows, but I'm not sure we do. In any case, delaying the update is a difference from a standard P2P bridge that could cause issues later.
When would you want the window assignment to occur? Directly when the registers containing the base/limit are being written to? There are multiple registers for base/limit, so I'll have to figure out a way to validate when the base/limit informations are valid.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 15 lines · Thomas Petazzoni
Quoted 7 lines
drivers/pci/host/pci-mvebu.c line 326
Quoted 3 lines
drivers/pci/host/pci-mvebu.c new file +3
-0+case PCI_PREF_LIMIT_UPPER32 : + bridge->preflimitupper = value; +break ;You're relying on a subsequent
PCI_COMMANDwrite to setPCI_COMMAND_IOand/orPCI_COMMAND_MEMORY, and you program the bridge windows at that time.Correct.
Quoted 4 lines
It might be a good idea if the PCI core did clear those bits
while updating the windows, but I'm not sure we do. In any case, delaying the update is a difference from a standard P2P bridge that could cause issues later.
Quoted 4 lines
When would you want the window assignment to occur? Directly when the registers containing the base/limit are being written to? There are multiple registers for base/limit, so I'll have to figure out a way to validate when the base/limit informations are valid.
For a standard bridge, the window change occurs immediately upon writing the base/limit, so that would be the best time.
Here's the scenario I have in mind: We have a bridge X leading to several devices A and B. Drivers are bound to A and B, and they are operating. A hotplug event requires us to expand or shrink one of X's windows, but the new window is still sufficient to operate A and B. The question is how we do the window update. If we turn off PCI_COMMAND_MEMORY for the update, then we have to coordinate with the drivers for A and B. But I think it is possible to do some updates safely even with the window decoding enabled, without any effect on A and B, and these are the ones that would cause trouble with your current scheme.
I don't think the PCI core does any updates like this today, but it's something we might like to do in the future.
Bjorn
PATCH 24/32 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 18 lines · Bjorn Helgaas
Quoted 15 lines · Thomas Petazzoni
Quoted 7 lines
drivers/pci/host/pci-mvebu.c line 326
Quoted 3 lines
drivers/pci/host/pci-mvebu.c new file +3
-0+case PCI_PREF_LIMIT_UPPER32 : + bridge->preflimitupper = value; +break ;You're relying on a subsequent
PCI_COMMANDwrite to setPCI_COMMAND_IOand/orPCI_COMMAND_MEMORY, and you program the bridge windows at that time.Correct.
Quoted 4 lines
It might be a good idea if the PCI core did clear those bits
while updating the windows, but I'm not sure we do. In any case, delaying the update is a difference from a standard P2P bridge that could cause issues later.
Quoted 7 lines
Quoted 4 lines
When would you want the window assignment to occur? Directly when the registers containing the base/limit are being written to? There are multiple registers for base/limit, so I'll have to figure out a way to validate when the base/limit informations are valid.
For a standard bridge, the window change occurs immediately upon writing the base/limit, so that would be the best time.
The standard says a window where limit <= base is invalid/disabled.
So the write mechanism from the PCI core should be
base = 0xFFFFFFFF
limit = true value
base = true value
In the driver the window should be synchronized after any base/limit update where base < limit. The initial values (ignoring fixed bits) should be
base = 0xFFFFFFFF
limit = 0
It would be optimal to use the resource framework to ensure the region is available, but that is something for the mbus driver to take care of, not PCI-E...
Quoted 4 lines
drivers for A and B. But I think it is possible to do some updates safely even with the window decoding enabled, without any effect on A and B, and these are the ones that would cause trouble with your current scheme.
A resize update should just write the true values in any order, the bridge window will expand without interrupting existing traffic.
However the core code must ensure that at no time do two bridges decode the same window. That could crash the machine..
Jason
PATCH 25/32 arm: mvebu: PCIe support is now available on mvebu
In reply to Thomas PetazzoniNow that the PCIe driver for mvebu has been integrated and all its relevant dependencies, we can mark the ARCH_MVEBU platform has MIGHT_HAVE_PCI, which allows to select the PCI bus support if needed.
--- arch/arm/mach-mvebu/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/arm/mach-mvebu/Kconfig +2 -0
@@ -13,6 +13,8 @@ config ARCH_MVEBU select MVEBU_CLK_CORE select MVEBU_CLK_CPU select MVEBU_CLK_GATING + select MIGHT_HAVE_PCI + select PCI_QUIRKS if PCI if ARCH_MVEBU
Signature · 1 line
1.7.9.5
PATCH 26/32 arm: mvebu: add PCIe Device Tree informations for Armada 370
In reply to Thomas PetazzoniThe Armada 370 SoC has two 1x PCIe 2.0 interfaces, so we add the necessary Device Tree informations to make these interfaces availabel.
--- arch/arm/boot/dts/armada-370.dtsi | 43 +++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/arm/boot/dts/armada-370.dtsi +43 -0
@@ -132,5 +132,48 @@ dmacap,memset; }; }; + + pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00001000 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 58>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 1,0 { + device_type = "pciex"; + reg = <0x1000 0 0xd0080000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 62>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 9>; + status = "disabled"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH 27/32 arm: mvebu: add PCIe Device Tree informations for Armada XP
In reply to Thomas PetazzoniThe Armada XP SoCs have multiple PCIe interfaces. The MV78230 has 2 PCIe units (one 4x or quad 1x, the other 1x only), the MV78260 has 3 PCIe units (two 4x or quad 1x and one 4x/1x), the MV78460 has 4 PCIe units (two 4x or quad 1x and two 4x/1x). We therefore add the necessary Device Tree informations to make those PCIe interfaces usable.
--- arch/arm/boot/dts/armada-xp-mv78230.dtsi | 99 ++++++++++++++++++ arch/arm/boot/dts/armada-xp-mv78260.dtsi | 114 ++++++++++++++++++++ arch/arm/boot/dts/armada-xp-mv78460.dtsi | 167 ++++++++++++++++++++++++++++++ 3 files changed, 380 insertions(+)
arch/arm/boot/dts/armada-xp-mv78230.dtsi +99 -0
@@ -70,5 +70,104 @@ #interrupts-cells = <2>; interrupts = <87>, <88>, <89>; }; + + /* + * MV78230 has 2 PCIe units Gen2.0: One unit can be + * configured as x4 or quad x1 lanes. One unit is + * x4/x1. + */ + pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 58>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 0,1 { + device_type = "pciex"; + reg = <0x1000 0 0xd0044000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 59>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <1>; + interrupts = <1>; + clocks = <&gateclk 6>; + status = "disabled"; + }; + + pcie at 0,2 { + device_type = "pciex"; + reg = <0x1800 0 0xd0048000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 60>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <2>; + interrupts = <1>; + clocks = <&gateclk 7>; + status = "disabled"; + }; + + pcie at 0,3 { + device_type = "pciex"; + reg = <0x2000 0 0xd004C000 0 0xC000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 61>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <3>; + interrupts = <1>; + clocks = <&gateclk 8>; + status = "disabled"; + }; + + pcie at 2,0 { + device_type = "pciex"; + reg = <0x4800 0 0xd0042000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &mpic 99>; + marvell,pcie-port = <2>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 26>; + status = "disabled"; + }; + }; }; };
arch/arm/boot/dts/armada-xp-mv78260.dtsi +114 -0
119 lines not on this page · raw message
arch/arm/boot/dts/armada-xp-mv78460.dtsi +167 -0
172 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 28/32 arm: mvebu: PCIe Device Tree informations for OpenBlocks AX3-4
In reply to Thomas PetazzoniThe PlatHome OpenBlocks AX3-4 has an internal mini-PCIe slot that can be used to plug mini-PCIe devices. We therefore enable the PCIe interface that corresponds to this slot.
--- arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts | 7 +++++++ 1 file changed, 7 insertions(+)
arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts +7 -0
@@ -121,5 +121,12 @@ nr-ports = <2>; status = "okay"; }; + pcie-controller { + status = "okay"; + /* Internal mini-PCIe connector */ + pcie at 0,0 { + status = "okay"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH 29/32 arm: mvebu: PCIe Device Tree informations for Armada XP DB
In reply to Thomas PetazzoniThe Marvell evaluation board (DB) for the Armada XP SoC has 6 physicals full-size PCIe slots, so we enable the corresponding PCIe interfaces in the Device Tree.
--- arch/arm/boot/dts/armada-xp-db.dts | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm/boot/dts/armada-xp-db.dts +27 -0
@@ -90,5 +90,32 @@ phy = <&phy3>; phy-mode = "sgmii"; }; + + pcie-controller { + status = "okay"; + + /* + * All 6 slots are physically present as + * standard PCIe slots on the board. + */ + pcie at 0,0 { + status = "okay"; + }; + pcie at 0,1 { + status = "okay"; + }; + pcie at 0,2 { + status = "okay"; + }; + pcie at 0,3 { + status = "okay"; + }; + pcie at 2,0 { + status = "okay"; + }; + pcie at 3,0 { + status = "okay"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH 30/32 arm: mvebu: PCIe Device Tree informations for Armada 370 Mirabox
In reply to Thomas PetazzoniThe Globalscale Mirabox platform uses one PCIe interface for an available mini-PCIe slot, and the other PCIe interface for an internal USB 3.0 controller. We add the necessary Device Tree informations to enable those two interfaces.
--- arch/arm/boot/dts/armada-370-mirabox.dts | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
arch/arm/boot/dts/armada-370-mirabox.dts +14 -0
@@ -52,5 +52,19 @@ phy = <&phy1>; phy-mode = "rgmii-id"; }; + + pcie-controller { + status = "okay"; + + /* Internal mini-PCIe connector */ + pcie at 0,0 { + status = "okay"; + }; + + /* Connected on the PCB to a USB 3.0 XHCI controller */ + pcie at 1,0 { + status = "okay"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH 31/32 arm: mvebu: PCIe Device Tree informations for Armada 370 DB
In reply to Thomas PetazzoniThe Marvell evaluation board (DB) for the Armada 370 SoC has 2 physical full-size PCIe slots, so we enable the corresponding PCIe interfaces in the Device Tree.
--- arch/arm/boot/dts/armada-370-db.dts | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/arm/boot/dts/armada-370-db.dts +15 -0
@@ -59,5 +59,20 @@ phy = <&phy1>; phy-mode = "rgmii-id"; }; + + pcie-controller { + status = "okay"; + /* + * The two PCIe units are accessible through + * both standard PCIe slots and mini-PCIe + * slots on the board. + */ + pcie at 0,0 { + status = "okay"; + }; + pcie at 1,0 { + status = "okay"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH 32/32 arm: mvebu: update defconfig with PCI and USB support
In reply to Thomas PetazzoniNow that we have the necessary drivers and Device Tree informations to support PCIe on Armada 370 and Armada XP, enable the CONFIG_PCI option.
Also, since the Armada 370 Mirabox has a built-in USB XHCI controller connected on the PCIe bus, enable the corresponding options as well.
--- arch/arm/configs/mvebu_defconfig | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm/configs/mvebu_defconfig +3 -1
@@ -13,6 +13,7 @@ CONFIG_MACH_ARMADA_370=y CONFIG_MACH_ARMADA_XP=y # CONFIG_CACHE_L2X0 is not set # CONFIG_SWP_EMULATE is not set +CONFIG_PCI=y CONFIG_SMP=y # CONFIG_LOCAL_TIMERS is not set CONFIG_AEABI=y
@@ -36,7 +37,8 @@ CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_DW=y CONFIG_GPIOLIB=y CONFIG_GPIO_SYSFS=y -# CONFIG_USB_SUPPORT is not set +CONFIG_USB=y +CONFIG_USB_XHCI_HCD=y CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_S35390A=y CONFIG_DMADEVICES=y
Signature · 1 line
1.7.9.5
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
In reply to Thomas PetazzoniQuoted 11 lines · Thomas Petazzoni
This series of patches introduces PCIe support for the Marvell Armada 370 and Armada XP.
This PATCHv3 follows:
* PATCHv2, sent on January, 28th 2013
* RFCv1, sent on December, 7th 2012Thanks to all the people who discussed on the previous version of the patch set. The discussions have been long and complicated, but certainly very useful.
Quoted 5 lines
In order to make progress on this, and help reducing the size of the patch set, I would ask if it would be possible to merge patches 1 to 17 from the series for 3.9 (only preparation work), keeping the rest for 3.10. The patches in question are PCI-related, ARM-related, and mvebu/orion-related.
You can add my 'Reviewed-by' tag on all paches except 5, 6, and 24.
I'm fine with adding the other ones to 3.9, as well as adding the binding document if you split that out from patch 24. Sorry for making this so hard for you.
Arnd
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
Dear Arnd Bergmann,
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines
In order to make progress on this, and help reducing the size of the patch set, I would ask if it would be possible to merge patches 1 to 17 from the series for 3.9 (only preparation work), keeping the rest for 3.10. The patches in question are PCI-related, ARM-related, and mvebu/orion-related.
You can add my 'Reviewed-by' tag on all paches except 5, 6, and 24.
Thanks. I would already be wonderful if I could merge many of those preparations patches for 3.9.
Quoted 3 lines
I'm fine with adding the other ones to 3.9, as well as adding the binding document if you split that out from patch 24. Sorry for making this so hard for you.
That's fine, the problem is difficult, so it's expected to be difficult to find the right solution.
However, I'd like to get serious alternate proposals when the current proposal isn't considered appropriate. A serious proposal needs to provide a mechanism to dynamically assign range of addresses to the different BARs of the different PCIe devices, and those range of addresses must comply with the special alignment requirements of the Marvell address decoding windows. So far, I haven't seen any alternate proposal that has those characteristics. The only alternate proposals were based on static assignments of addresses to each PCIe interface, which is unacceptable.
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
In reply to Arnd BergmannDear Arnd Bergmann,
Quoted 5 lines · Arnd Bergmann
You can add my 'Reviewed-by' tag on all paches except 5, 6, and 24.
I'm fine with adding the other ones to 3.9, as well as adding the binding document if you split that out from patch 24. Sorry for making this so hard for you.
Is this really true that some more code can be taken for 3.9 ? It somewhat contradicts the statement that Olof has made on LAKML just a few hours ago "arm-soc: No new material from 3.9, fixes only".
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
Quoted 11 lines · Thomas Petazzoni
Quoted 5 lines · Arnd Bergmann
You can add my 'Reviewed-by' tag on all paches except 5, 6, and 24.
I'm fine with adding the other ones to 3.9, as well as adding the binding document if you split that out from patch 24. Sorry for making this so hard for you.
Is this really true that some more code can be taken for 3.9 ? It somewhat contradicts the statement that Olof has made on LAKML just a few hours ago "arm-soc: No new material from 3.9, fixes only".
Sorry, that was bad communiation between Olof and me. Olof has been handling basically all of the pulls for this release and his decision is final. The only part of the series that I believe we should take is patch 7, since that is a bug fix.
Arnd
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
In reply to Thomas PetazzoniHi Thomas,
Quoted 2 lines · Thomas Petazzoni
This series of patches introduces PCIe support for the Marvell Armada 370 and Armada XP.
Tested on a Marvell DB-MV784MP-GP evaluation board with a Marvell 9220 SATA controller. Disks attached to this controller are working.
# /usr/sbin/lspci -v -s 01:00.0
01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9220 (rev 10) (prog-if 01 [AHCI 1.0])
Subsystem: Marvell Technology Group Ltd. Device 9220
Flags: bus master, fast devsel, latency 0, IRQ 105
I/O ports at 10020 [size=8]
I/O ports at 10030 [size=4]
I/O ports at 10028 [size=8]
I/O ports at 10034 [size=4]
I/O ports at 10000 [size=32]
Memory at c1000000 (32-bit, non-prefetchable) [size=2K]
Expansion ROM at c1100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [70] Express Legacy Endpoint, MSI 00
Capabilities: [e0] SATA HBA v0.0
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: ahciQuoted 371 lines
This PATCHv3 follows:
* PATCHv2, sent on January, 28th 2013
* RFCv1, sent on December, 7th 2012Thanks to all the people who discussed on the previous version of the patch set. The discussions have been long and complicated, but certainly very useful.
In order to make progress on this, and help reducing the size of the patch set, I would ask if it would be possible to merge patches 1 to 17 from the series for 3.9 (only preparation work), keeping the rest for 3.10. The patches in question are PCI-related, ARM-related, and mvebu/orion-related.
Changes between v2 and v3:
* Use
of_irq_map_pci()instead ofof_irq_map_raw(), as suggested by Andrew Murray. In order to do this, we moved the interrupt-map and interrupt-map-mask DT properties from the main PCIe controller node to the DT subnodes representing each PCIe interface.* Remove the usage of the emulated host bridge.
* Move the emulated PCI-to-PCI bridge code into the Marvell PCI driver itself, in order to allow a tighter integration. Suggested by Bjorn Helgaas and Jason Gunthorpe.
* Make the allocation of address decoding windows dynamic: it's when memory accesses or I/O accesses are enabled at the PCI-to-PCI bridge level that we allocate and setup the corresponding address decoding window. Requested by Bjorn Helgaas.
* Fixed the implementation of I/O accesses to use I/O addresses that fall within the normal
IO_SPACE_LIMIT. This required using the "remap" functionality of address decoding windows, and therefore some changes in the address decoding window allocator. Follows a long discussion about I/O accesses.* Set up a correct bus number in the configuration of the PCIe interfaces so that we don't have to fake bus numbers anymore. Requested by Jason Gunthorpe.
* Fix the
of_pci_get_devfn()implementation according to Stephen Warren's comment.* Use
CFLAGS_instead of ccflags to add the mach-mvebu and plat-orion include paths when building the pci-mvebu driver. This ensures that the include paths are only added when building this specific driver. Requested by Stephen Warren.* Fix the ->resource_align() to only apply on bus 0 (the one on which the emulated PCI-to-PCI bridges sit), and to request an alignment on the size of the window (and not only 64 KB for I/O windows and 1 MB for memory windows).
* Clarified the commit log of "clk: mvebu: create parent-child relation for PCIe clocks on Armada 370"
A quick description of the patches:
* Patches 1 to 3 add PCI-related Device Tree parsing functions. Those patches are common with the Nvidia Tegra PCIe patch set from Thierry Redding. They are included in this series so that it can be tested easily.
* Patch 4 extends the ARM PCI core to store a per-controller private data pointer. This patch is common with the Nvidia Tegra PCIe patch set from Thierry Redding. It is included in this series so that it can be tested easily.
* Patch 5 fixes a problem in
lib/devres.cthat prevents certain PCI-related functions from being visible onNO_IOPORTplatforms. I know this patch isn't acceptable by itself, but the discussion about this has been so huge and went in so many directions that in the end, I don't know what is the correct way of fixing this. If an agreement is found on how to fix this properly, I'm willing to work on it if needed.* Patch 6 extends the ARM PCI core with an additional hook that a PCI controller driver can register and get called to realign PCI ressource addresses. This is needed for the support of Marvell PCIe interfaces because the address decoding windows for I/O ranges have a granularity of 64 KB, while the PCI standard requires only a 4 KB alignement. See the patch itself for details.
* Patch 7 fixes a mistake in the interrupt controller node of the Armada 370/XP Device Tree, which was invisible until we started using the
of_irq_map_raw()function, needed in our PCIe support.* Patches 8 and 9 fix some issues in the Armada 370/XP clock gating driver, related to PCIe interfaces.
* Patches 10 and 11 are cleanup/refactoring of the common plat-orion address decoding code, in preparation for further changes related to PCIe.
* Patches 12 to 17 introduce a
ORION_ADDR_MAP_NO_REMAPdefine that is used by existing Marvell SoC code to say "I don't need this window to remap anything". Previously a -1 value was used as the remap address to communicate the fact that no remap is needed, but this prevents any remap address higher than 2 GB.* Patch 18 removes
__initfrom a few address window decoding functions that are now needed after boot.* Patch 19 introduces in the common plat-orion address decoding code functions to allocate/free an address decoding window. Until now, the address decoding windows were configured statically. With Armada XP having up to 10 PCIe interfaces, we don't want to allocate useless address decoding windows statically, so we move to a more dynamic model in which address decoding windows are configured only for the PCIe interfaces that are actually in use.
* Patch 20 removes
__initfrom a few PCIe functions that are now needed after boot.* Patch 21 improves the Armada 370/XP specific address decoding code to provide functions that add and remove an address decoding window for a given PCIe interface. It relies on the common functions added in patch 19.
* Patch 22 makes the common plat-orion PCIe code available on
PLAT_ORIONplatforms such asARCH_MVEBU.* Patch 23 creates the
drivers/pci/hostdirectory and makes the related minimal changes to Kconfig/Makefile. This patch will trivially conflict with the NVidia Tegra PCIe support posted by Thierry Redding, which also creates thedrivers/pci/hostdirectory.* Patch 24 contains the Armada 370/XP PCIe driver itself, that implements the necessary operations required by the ARM PCI core, and configures the address decoding windows as needed. This driver relies on a Device Tree description of the PCIe interfaces.
* Patch 25 marks the
ARCH_MVEBUplatform has having PCI available, which allows the compilation of the PCIe support.* Patches 26 and 27 add the SoC-level Device Tree informations related to PCIe for Armada 370 and Armada XP.
* Patch 28 to 31 add the board-level Device Tree informations related to PCIe for the Armada XP DB, Armada 370 DB, PlatHome OpenBlocks AX3-4 and GlobalScale Mirabox boards.
* Patch 32 updates
mvebu_defconfigwith PCI and USB support.This patch set applies on top of v3.8-rc7, and has been pushed at:
git://github.com/MISL-EBU-System-SW/mainline-public.gitmarvell-pcie-v3Thanks,
Thomas
---
Output of lspci -vvv:00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00010000-00010fff
Memory behind bridge: c1000000-c10fffff
Prefetchable memory behind bridge: c1100000-c11fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: 00020000-00020fff
Memory behind bridge: c1200000-c12fffff
Prefetchable memory behind bridge: c1300000-c13fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 7846 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>03:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
Subsystem: Marvell Technology Group Ltd. Device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 105
Region 0: Memory atc1000000(64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at 10000 [size=256]
[virtual] Expansion ROM atc1100000[disabled] [size=512K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000012345678 Data: 0000
Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <256ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 <256ns, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use:sata_mv05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 106
Region 0: Memory atc1200000(32-bit, non-prefetchable) [size=128K]
Region 1: Memory atc1220000(32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 20000 [disabled] [size=32]
[virtual] Expansion ROM atc1300000[disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mAPME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-c1-c4-fe
Kernel driver in use: e1000eBoot messages from the PCI subsystem:
mvebu-pcie pcie-controller.1: PCIe0.0: link down
mvebu-pcie pcie-controller.1: PCIe0.1: link down
mvebu-pcie pcie-controller.1: PCIe0.2: link up
mvebu-pcie pcie-controller.1: PCIe0.3: link down
mvebu-pcie pcie-controller.1: PCIe2.0: link up
mvebu-pcie pcie-controller.1: PCIe3.0: link down
mvebu-pcie pcie-controller.1: PCI host bridge to bus 0000:00pci_bus0000:00: root bus resource [io 0x0000-0xfffff]pci_bus0000:00: root bus resource [mem 0xc1000000-0xc8ffffff]pci_bus0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers enabled
PCI: bus2: Fast back to back transfers enabled
PCI: bus3: Fast back to back transfers disabled
PCI: bus4: Fast back to back transfers enabled
PCI: bus5: Fast back to back transfers disabled
PCI: bus6: Fast back to back transfers enabled
pci 0000:00:03.0: BAR 8: assigned [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: BAR 9: assigned [mem 0xc1100000-0xc11fffff pref] pci 0000:00:05.0: BAR 8: assigned [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: BAR 9: assigned [mem 0xc1300000-0xc13fffff pref] pci 0000:00:03.0: BAR 7: assigned [io 0x10000-0x10fff]
pci 0000:00:05.0: BAR 7: assigned [io 0x20000-0x20fff]
pci 0000:00:01.0: PCI bridge to [bus 01]
pci 0000:00:02.0: PCI bridge to [bus 02]
pci 0000:03:00.0: BAR 0: assigned [mem 0xc1000000-0xc10fffff 64bit] pci 0000:03:00.0: BAR 6: assigned [mem 0xc1100000-0xc117ffff pref] pci 0000:03:00.0: BAR 2: assigned [io 0x10000-0x100ff]
pci 0000:00:03.0: PCI bridge to [bus 03]pci 0000:00:03.0: bridge window [io 0x10000-0x10fff] pci 0000:00:03.0: bridge window [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: bridge window [mem 0xc1100000-0xc11fffff pref]pci 0000:00:04.0: PCI bridge to [bus 04]
pci 0000:05:00.0: BAR 0: assigned [mem 0xc1200000-0xc121ffff] pci 0000:05:00.0: BAR 1: assigned [mem 0xc1220000-0xc123ffff] pci 0000:05:00.0: BAR 6: assigned [mem 0xc1300000-0xc131ffff pref] pci 0000:05:00.0: BAR 2: assigned [io 0x20000-0x2001f] pci 0000:00:05.0: PCI bridge to [bus 05]pci 0000:00:05.0: bridge window [io 0x20000-0x20fff] pci 0000:00:05.0: bridge window [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: bridge window [mem 0xc1300000-0xc13fffff pref]pci 0000:00:06.0: PCI bridge to [bus 06]
PCI: enabling device 0000:00:01.0 (0140 -> 0143)
PCI: enabling device 0000:00:02.0 (0140 -> 0143)
PCI: enabling device 0000:00:03.0 (0140 -> 0143)
PCI: enabling device 0000:00:04.0 (0140 -> 0143)
PCI: enabling device 0000:00:05.0 (0140 -> 0143)
PCI: enabling device 0000:00:06.0 (0140 -> 0143)_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Signature · 1 line
Christophe Vu-Brugier
PATCH v3 1/1 PCIe support for the Armada 370 and Armada XP SoCs
Dear Christophe Vu-Brugier,
Quoted 10 lines · Christophe Vu-Brugier
Hi Thomas,
Quoted 2 lines · Thomas Petazzoni
This series of patches introduces PCIe support for the Marvell Armada 370 and Armada XP.
Tested on a Marvell DB-MV784MP-GP evaluation board with a Marvell 9220 SATA controller. Disks attached to this controller are working.
Awesome, thanks a lot again Christophe for having taken the time to test this patch series!
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com