PCIe support for the Armada 370 and Armada XP SoCs
Subsystems (10)
PATCH v2 1/1 PCIe support for the Armada 370 and Armada XP SoCs
Hello,
This series of patches introduces PCIe support for the Marvell Armada 370 and Armada XP. This "PATCH v2" follows the "RFC v1" that has been sent on December, 7th. If possible, I'd like this patch series to be considered 3.9 material, so reviews and comments are highly appreciated in order to converge towards something that can be merged.
Since the previous version, the main changes are:
* We now use the standard PCI Device Tree bindings, both for describing memory ranges, and interrupts.
* We now use an emulated PCI host bridge and emulated PCI-to-PCI bridges to link together the PCIe interfaces of the SoC. This also has the added benefit that the allocation of memory and I/O addresses is now done by the PCI core, which avoids oversizing address decoding windows and removes a specific PCI address allocator that was present in the v1.
* The driver is now located in drivers/pci/host.
On the Armada XP evaluation board that has 6 usable PCIe slots, one Intel e1000e PCIe card connected in slot 3 and one Marvell SATA PCIe card connected in slot 4, the lspci -t output is:
-[0000:00]-+-00.0
+-01.0-[01]--
+-02.0-[02]--
+-03.0-[03]----00.0
+-04.0-[04]----00.0
+-05.0-[09]--
\-06.0-[0a]--With lspci showing:
00:00.0 Host bridge: Marvell Technology Group Ltd. Device 102d
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 1092
03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) 04:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
Please see at the end of this e-mail the entire output of lspci -vvv.
A quick description of the patches:
* Patches 1 to 3 add PCI-related Device Tree parsing functions. Those patches are common with the Nvidia Tegra PCIe patch set from Thierry Redding. They are included in this series so that it can be tested easily.
* Patch 4 extends the ARM PCI core to store a per-controller private data pointer. This patch is common with the Nvidia Tegra PCIe patch set from Thierry Redding. It is included in this series so that it can be tested easily.
* Patch 5 extends the ARM PCI core with an additional hook that a PCI controller driver can register and get called to realign PCI ressource addresses. This is needed for the support of Marvell PCIe interfaces because the address decoding windows for I/O ranges have a granularity of 64 KB, while the PCI standard requires only a 4 KB alignement. See the patch itself for details.
* Patch 6 fixes a problem in lib/devres.c that prevents certain PCI-related functions from being visible on NO_IOPORT platforms. I know this patch isn't acceptable by itself, but the discussion about this has been so huge and went in so many directions that in the end, I don't know what is the correct way of fixing this. If an agreement is found on how to fix this properly, I'm willing to work on it if needed.
* Patch 7 and 8 respectively add a minimal emulated PCI host bridge and emulated PCI-to-PCI bridge. The former has originally been written by Thierry Redding. Even though I made a few minor changes to it, I kept Thierry's authorship on it (Thierry do not hesitate to let me know if my changes are too substantial to keep your name on this code). The latter was largely inspired by Thierry's code.
* Patch 9 creates the drivers/pci/host directory and makes the related minimal changes to Kconfig/Makefile. This patch will trivially conflict with the NVidia Tegra PCIe support posted by Thierry Redding, which also creates the drivers/pci/host directory.
* Patch 10 fixes a mistake in the interrupt controller node of the Armada 370/XP Device Tree, which was invisible until we started using the of_irq_map_raw() function, needed in our PCIe support.
* Patch 11 and 12 fixes some issues in the Armada 370/XP clock gating driver, related to PCIe interfaces.
* Patch 13 and 14 are cleanup/refactoring of the common plat-orion address decoding code, in preparation for further changes related to PCIe.
* Patch 15 introduces in the common plat-orion address decoding code functions to allocate/free an address decoding window. Until now, the address decoding windows were configured statically. With Armada XP having up to 10 PCIe interfaces, we don't want to allocate useless address decoding windows statically, so we move to a more dynamic model in which address decoding windows are configured only for the PCIe interfaces that are actually in use.
* Patch 16 improves the Armada 370/XP specific address decoding code to provide functions that add and remove an address decoding window for a given PCIe interface. It relies on the common functions added in patch 15.
* Patch 17 makes the common plat-orion PCIe code available on PLAT_ORION platforms such as ARCH_MVEBU.
* Patch 18 adds some variants of the PCI configuration space read/write functions that were available in the plat-orion/pcie.c code. Those variants are needed due to the usage of the emulated host-bridge + emulated PCI-to-PCI bridge model.
* Patch 19 contains the Armada 370/XP PCIe driver itself, that implements the necessary operations required by the ARM PCI core, and configures the address decoding windows as needed. This driver relies on a Device Tree description of the PCIe interfaces.
* Patch 20 marks the ARCH_MVEBU platform has having PCI available, which allows the compilation of the PCIe support.
* Patches 21 and 22 add the SoC-level Device Tree informations related to PCIe for Armada 370 and Armada XP.
* Patch 23, 24, 25 and 26 add the board-level Device Tree informations related to PCIe for the Armada XP DB, Armada 370 DB, PlatHome OpenBlocks AX3-4 and GlobalScale Mirabox boards.
* Patch 27 updates mvebu_defconfig with PCI and USB support.
This patch set applies on top of v3.8-rc5, and has been pushed at:
git://github.com/MISL-EBU-System-SW/mainline-public.git marvell-pcie-v2
Thanks,
Thomas
---
Output of lspci -vvv:
# /usr/sbin/lspci -vvv 00:00.0 Host bridge: Marvell Technology Group Ltd. Device 102d Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin ? routed to IRQ 0 Capabilities: [fc] <chain broken>
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: c0000000-c0000fff
Memory behind bridge: c1000000-c10fffff
Prefetchable memory behind bridge: c1100000-c11fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: c0010000-c0010fff
Memory behind bridge: c1200000-c12fffff
Prefetchable memory behind bridge: c1300000-c13fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=09, subordinate=09, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort+ >Reset+ FastB2B+
PriDiscTmr+ SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn+
Capabilities: [fc] <chain broken>
03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 105
Region 0: Memory at c1000000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at c1020000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at c0000000 [disabled] [size=32]
[virtual] Expansion ROM@c1100000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-c1-c4-fe
Kernel driver in use: e1000e
04:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
Subsystem: Marvell Technology Group Ltd. Device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 106
Region 0: Memory at c1200000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at c0010000 [size=256]
[virtual] Expansion ROM@c1300000 [disabled] [size=512K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000012345678 Data: 0000
Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <256ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 <256ns, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: sata_mv
PATCH v2 1/27 of/pci: Provide support for parsing PCI DT ranges property
From: Andrew Murray <[email protected]>
DT bindings for PCI host bridges often use the ranges property to describe memory and IO ranges - this binding tends to be the same across architectures yet several parsing implementations exist, e.g. arch/mips/pci/pci.c, arch/powerpc/kernel/pci-common.c, arch/sparc/kernel/pci.c and arch/microblaze/pci/pci-common.c (clone of PPC). Some of these duplicate functionality provided by drivers/of/address.c.
This patch provides a common iterator-based parser for the ranges property, it is hoped this will reduce DT representation differences between architectures and that architectures will migrate in part to this new parser.
It is also hoped (and the motativation for the patch) that this patch will reduce duplication of code when writing host bridge drivers that are supported by multiple architectures.
This patch provides struct resources from a device tree node, e.g.:
u32 *last = NULL;
struct resource res;
while ((last = of_pci_process_ranges(np, res, last))) {
//do something with res
}Platforms with quirks can then do what they like with the resource or migrate common quirk handling to the parser. In an ideal world drivers can just request the obtained resources and pass them on (e.g. pci_add_resource_offset).
--- drivers/of/address.c | 63 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/of_address.h | 9 +++++++ 2 files changed, 72 insertions(+)
drivers/of/address.c +63 -0
@@ -13,6 +13,7 @@#define OF_CHECK_COUNTS (na, ns) (OF_CHECK_ADDR_COUNT(na) && (ns) > 0)static struct of_bus *of_match_bus (struct device_node *np); +static struct of_bus *of_find_bus (const char *name);static int __of_address_to_resource (struct device_node *dev,const __be32 *addrp,u64 size,unsigned int flags,const char *name,struct resource *r);
@@ -227,6 +228,57 @@ int of_pci_address_to_resource(struct device_node *dev, int bar,return __of_address_to_resource (dev, addrp, size, flags,NULL , r); }EXPORT_SYMBOL_GPL (of_pci_address_to_resource); + +const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res,const __be32 *from) +{ +const __be32 *start, *end; +int na, ns, np, pna; +int rlen; +struct of_bus *bus; + +WARN_ON (!res); + + bus =of_find_bus ("pci" ); + bus->count_cells (node, &na, &ns); +if (!OF_CHECK_COUNTS (na, ns)) { +pr_err ("Bad cell count for %s\n" , node->full_name); +return NULL ; + } + + pna =of_n_addr_cells (node); + np = pna + na + ns; + + start =of_get_property (node,"ranges" , &rlen); +if (start ==NULL ) +return NULL ; + + end = start + rlen /sizeof (__be32); + +if (!from) + from = start; + +while (from + np <= end) { +u64 cpu_addr, size; + + cpu_addr =of_translate_address (node, from + na); + size =of_read_number (from + na + pna, ns); + res->flags = bus->get_flags (from); + from += np; + +if (cpu_addr ==OF_BAD_ADDR || size ==0 ) +continue ; + + res->name = node->full_name; + res->start = cpu_addr; + res->end = res->start + size -1 ; + res->parent = res->child = res->sibling =NULL ; +return from; + } + +return NULL ; +} +EXPORT_SYMBOL_GPL (of_pci_process_ranges);#endif /* CONFIG_PCI */ /*
@@ -337,6 +389,17 @@ static struct of_bus *of_match_bus(struct device_node *np)return NULL ; } +static struct of_bus *of_find_bus (const char *name) +{ +unsigned int i; + +for (i =0 ; i <ARRAY_SIZE (of_busses); i++) +if (strcmp (name, of_busses[i].name) ==0 ) +return &of_busses[i]; + +return NULL ; +} +static int of_translate_one (struct device_node *parent,struct of_bus *bus,struct of_bus *pbus,__be32 *addr,int na,int ns,int pna,const char *rprop)
include/linux/of_address.h +9 -0
@@ -27,6 +27,8 @@ static inline unsigned long pci_address_to_pio(phys_addr_t addr) { return -1; }#define pci_address_to_pio pci_address_to_pio#endif +const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res,const __be32 *from);#else /* CONFIG_OF_ADDRESS */ #ifndef of_address_to_resource static inline int of_address_to_resource(struct device_node *dev, int index,
@@ -53,6 +55,13 @@ static inline const __be32 *of_get_address(struct device_node *dev, int index, {return NULL ; } + +static inline const __be32 *of_pci_process_ranges (struct device_node *node, +struct resource *res, +const __be32 *from) +{ +return NULL ; +}#endif /* CONFIG_OF_ADDRESS */
Signature · 1 line
1.7.9.5
PATCH v2 2/27 of/pci: Add of_pci_get_devfn() function
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
This function can be used to parse the device and function number from a standard 5-cell PCI resource. PCI_SLOT() and PCI_FUNC() can be used on the returned value obtain the device and function numbers respectively.
--- drivers/of/of_pci.c | 32 ++++++++++++++++++++++++++++---- include/linux/of_pci.h | 1 + 2 files changed, 29 insertions(+), 4 deletions(-)
drivers/of/of_pci.c +28 -4
@@ -7,12 +7,13 @@static inline int __of_pci_pci_compare (struct device_node *node,unsigned int devfn) { -unsigned int size; -const __be32 *reg =of_get_property (node,"reg" , &size); +int err; -if (!reg || size <5 *sizeof (__be32)) + err =of_pci_get_devfn (node); +if (err <0 )return 0 ; -return ((be32_to_cpup (®[0 ]) >>8 ) &0xff ) == devfn; + +return devfn == err; } struct device_node *of_pci_find_child_device(struct device_node *parent,
@@ -40,3 +41,26 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,return NULL ; }EXPORT_SYMBOL_GPL (of_pci_find_child_device); + +/** +* of_pci_get_devfn() - Get device and function numbers for a device node +* @np: device node +* +* Parses a standard 5-cell PCI resource and returns an 8-bit value that can +* be passed to the PCI_SLOT() and PCI_FUNC() macros to extract the device +* and function numbers respectively. On error a negative error code is +* returned. +*/ +int of_pci_get_devfn (struct device_node *np) +{ +unsigned int size; +const __be32 *reg; + + reg =of_get_property (np,"reg" , &size); + +if (!reg || size <5 *sizeof (__be32)) +return -EINVAL ; + +return (be32_to_cpup (reg) >>8 ) &0xff ; +} +EXPORT_SYMBOL_GPL (of_pci_get_devfn);
include/linux/of_pci.h +1 -0
@@ -10,5 +10,6 @@ int of_irq_map_pci(const struct pci_dev *pdev, struct of_irq *out_irq);struct device_node ;struct device_node *of_pci_find_child_device (struct device_node *parent,unsigned int devfn); +int of_pci_get_devfn (struct device_node *np);#endif
Signature · 1 line
1.7.9.5
PATCH v2 2/27 of/pci: Add of_pci_get_devfn() function
Quoted 7 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
This function can be used to parse the device and function number from a standard 5-cell PCI resource.
PCI_SLOT()andPCI_FUNC()can be used on the returned value obtain the device and function numbers respectively.
drivers/of/of_pci.c line 10
Quoted 7 lines
drivers/of/of_pci.c +1
-2static inline int __of_pci_pci_compare (struct device_node *node,unsigned int devfn) { -unsigned int size; -const __be32 *reg =of_get_property (node,"reg" , &size); +int err;
I think I commented when Thierry posted this, that calling that "err" seems a little odd. Thierry replied:
Maybe renaming the devfn parameter to data and using devfn for the local variable would be more obvious.
PATCH v2 2/27 of/pci: Add of_pci_get_devfn() function
Quoted 19 lines · Stephen Warren
Quoted 7 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
This function can be used to parse the device and function number from a standard 5-cell PCI resource.
PCI_SLOT()andPCI_FUNC()can be used on the returned value obtain the device and function numbers respectively.drivers/of/of_pci.c line 10
Quoted 7 lines
drivers/of/of_pci.c +1
-2static inline int __of_pci_pci_compare (struct device_node *node,unsigned int devfn) { -unsigned int size; -const __be32 *reg =of_get_property (node,"reg" , &size); +int err;I think I commented when Thierry posted this, that calling that "err" seems a little odd. Thierry replied:
Quoted 2 lines
Maybe renaming the devfn parameter to data and using devfn for the local variable would be more obvious.
That's already fixed up in my series. I was going to wait until I was done with the MSI rework but maybe posting an intermediate version is in order to share the latest state.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130128/02aacdbf/attachment.sig>
PATCH v2 2/27 of/pci: Add of_pci_get_devfn() function
Thierry, Stephen,
Quoted 3 lines · Thierry Reding
That's already fixed up in my series. I was going to wait until I was done with the MSI rework but maybe posting an intermediate version is in order to share the latest state.
Thanks! Those first four patches are definitely shared between our series.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 3/27 of/pci: Add of_pci_parse_bus_range() function
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
This function can be used to parse a bus-range property as specified by device nodes representing PCI bridges.
--- drivers/of/of_pci.c | 25 +++++++++++++++++++++++++ include/linux/of_pci.h | 1 + 2 files changed, 26 insertions(+)
drivers/of/of_pci.c +25 -0
@@ -64,3 +64,28 @@ int of_pci_get_devfn(struct device_node *np)return (be32_to_cpup (reg) >>8 ) &0xff ; }EXPORT_SYMBOL_GPL (of_pci_get_devfn); + +/** +* of_pci_parse_bus_range() - parse the bus-range property of a PCI device +* @node: device node +* @res: address to a struct resource to return the bus-range +* +* Returns 0 on success or a negative error-code on failure. +*/ +int of_pci_parse_bus_range (struct device_node *node,struct resource *res) +{ +const __be32 *values; +int len; + + values =of_get_property (node,"bus-range" , &len); +if (!values || len <sizeof (*values) *2 ) +return -EINVAL ; + + res->name = node->name; + res->start =be32_to_cpup (values++); + res->end =be32_to_cpup (values); + res->flags =IORESOURCE_BUS ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (of_pci_parse_bus_range);
include/linux/of_pci.h +1 -0
@@ -11,5 +11,6 @@ struct device_node;struct device_node *of_pci_find_child_device (struct device_node *parent,unsigned int devfn);int of_pci_get_devfn (struct device_node *np); +int of_pci_parse_bus_range (struct device_node *node,struct resource *res);#endif
Signature · 1 line
1.7.9.5
PATCH v2 4/27 ARM: pci: Allow passing per-controller private data
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
In order to allow drivers to specify private data for each controller, this commit adds a private_data field to the struct hw_pci. This field is an array of nr_controllers pointers that will be used to initialize the private_data field of the corresponding controller's pci_sys_data structure.
--- arch/arm/include/asm/mach/pci.h | 1 + arch/arm/kernel/bios32.c | 3 +++ 2 files changed, 4 insertions(+)
arch/arm/include/asm/mach/pci.h +1 -0
@@ -23,6 +23,7 @@ struct hw_pci {#endif struct pci_ops *ops;int nr_controllers; +void **private_data;int (*setup)(int nr,struct pci_sys_data *);struct pci_bus *(*scan)(int nr,struct pci_sys_data *);void (*preinit)(void );
arch/arm/kernel/bios32.c +3 -0
@@ -464,6 +464,9 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->map_irq = hw->map_irq;INIT_LIST_HEAD (&sys->resources); +if (hw->private_data) + sys->private_data = hw->private_data[nr]; + ret = hw->setup (nr, sys);if (ret >0 ) {
Signature · 1 line
1.7.9.5
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniThe PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device). For PCIe memory windows, those windows are defined with a 1 MB granularity (which matches the PCI specs), but PCIe I/O windows can only be defined with a 64 KB granularity, so they have to be 64 KB aligned. We therefore need to tell the PCI core about this special alignement requirement.
The PCI core already calls pcibios_align_resource() in the ARM PCI core, specifically for such purposes. So this patch extends the ARM PCI core so that it calls a ->align_resource() hook registered by the PCI driver, exactly like the existing ->map_irq() and ->swizzle() hooks.
A particular PCI driver can register a align_resource() hook, and do its own specific alignement, depending on the specific constraints of the underlying hardware.
--- arch/arm/include/asm/mach/pci.h | 11 +++++++++++ arch/arm/kernel/bios32.c | 6 ++++++ 2 files changed, 17 insertions(+)
arch/arm/include/asm/mach/pci.h +11 -0
@@ -30,6 +30,11 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align); };/*
@@ -51,6 +56,12 @@ struct pci_sys_data {u8 (*swizzle)(struct pci_dev *,u8 *);/* IRQ mapping */ int (*map_irq)(const struct pci_dev *,u8 ,u8 ); +/* Resource alignement requirements */ +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align);void *private_data;/* platform controller private data */ };
arch/arm/kernel/bios32.c +6 -0
@@ -462,6 +462,7 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->busnr = busnr; sys->swizzle = hw->swizzle; sys->map_irq = hw->map_irq; + sys->align_resource = hw->align_resource;INIT_LIST_HEAD (&sys->resources);if (hw->private_data)
@@ -574,6 +575,8 @@ char * __init pcibios_setup(char *str)resource_size_t pcibios_align_resource (void *data,const struct resource *res,resource_size_t size,resource_size_t align) { +struct pci_dev *dev = data; +struct pci_sys_data *sys = dev->sysdata;resource_size_t start = res->start;if (res->flags &IORESOURCE_IO && start &0x300 )
@@ -581,6 +584,9 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res, start = (start + align -1 ) & ~(align -1 ); +if (sys->align_resource) +return sys->align_resource (dev, res, start, size, align); +return start; }
Signature · 1 line
1.7.9.5
PATCH v2 5/27 arm: pci: add a align_resource hook
Russell,
As the arch/arm/kernel/ maintainer, what is your position regarding the below patch?
Thanks for your review,
Thomas
Quoted 88 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
However, the Marvell PCIe interfaces rely on address decoding windows (which allow to associate a range of physical addresses with a given device). For PCIe memory windows, those windows are defined with a 1 MB granularity (which matches the PCI specs), but PCIe I/O windows can only be defined with a 64 KB granularity, so they have to be 64 KB aligned. We therefore need to tell the PCI core about this special alignement requirement.
The PCI core already calls
pcibios_align_resource()in the ARM PCI core, specifically for such purposes. So this patch extends the ARM PCI core so that it calls a ->align_resource() hook registered by the PCI driver, exactly like the existing ->map_irq() and ->swizzle() hooks.A particular PCI driver can register a
align_resource()hook, and do its own specific alignement, depending on the specific constraints of the underlying hardware.Signed-off-by Thomas Petazzoni <[email protected]>Cc Russell King <[email protected]>--- arch/arm/include/asm/mach/pci.h | 11 +++++++++++ arch/arm/kernel/bios32.c | 6 ++++++ 2 files changed, 17 insertions(+)arch/arm/include/asm/mach/pci.h +11
-0@@ -30,6 +30,11 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align); };/* @@ -51,6 +56,12 @@ struct pci_sys_data {u8 (*swizzle)(struct pci_dev *,u8 *);/* IRQ mapping */ int (*map_irq)(const struct pci_dev *,u8 ,u8 ); +/* Resource alignement requirements */ +resource_size_t (*align_resource)(struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align);void *private_data;/* platform controller private data */ };arch/arm/kernel/bios32.c +6
-0@@ -462,6 +462,7 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->busnr = busnr; sys->swizzle = hw->swizzle; sys->map_irq = hw->map_irq; + sys->align_resource = hw->align_resource;INIT_LIST_HEAD (&sys->resources);if (hw->private_data)@@ -574,6 +575,8 @@ char * __init pcibios_setup(char *str)resource_size_t pcibios_align_resource (void *data,const struct resource *res,resource_size_t size,resource_size_t align) { +struct pci_dev *dev = data; +struct pci_sys_data *sys = dev->sysdata;resource_size_t start = res->start;if (res->flags &IORESOURCE_IO && start &0x300 )@@ -581,6 +584,9 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res, start = (start + align -1 ) & ~(align -1 ); +if (sys->align_resource) +return sys->align_resource (dev, res, start, size, align); +return start; }
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 4 lines · Thomas Petazzoni
Russell,
As the
arch/arm/kernel/maintainer, what is your position regarding the below patch?
Given the description, I'd feel much happier with this if we specified the alignment numerically rather than allowing "some random code" to do something with the passed values.
PATCH v2 5/27 arm: pci: add a align_resource hook
Russell,
Thanks for your quick feedback!
Quoted 3 lines · Russell King - ARM Linux
Given the description, I'd feel much happier with this if we specified the alignment numerically rather than allowing "some random code" to do something with the passed values.
So, you'd prefer to have two new members added in the hw_pci structure to give the alignment requirements for I/O regions and memory regions?
Something like:
struct hw_pci {
[...]
unsigned long io_align;
unsigned long mem_align;
};If that's fine with you, I'll go ahead and change the implementation in this direction. As long as I can express my special 64 KB alignment requirement for I/O regions, I'm just fine :-)
Thanks again,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 4 lines · Thomas Petazzoni
Russell,
Thanks for your quick feedback!
Quoted 8 lines
Quoted 3 lines · Russell King - ARM Linux
Given the description, I'd feel much happier with this if we specified the alignment numerically rather than allowing "some random code" to do something with the passed values.
So, you'd prefer to have two new members added in the
hw_pcistructure to give the alignment requirements for I/O regions and memory regions?
Yep, otherwise we'll have yet more code to review rather than one algorithm with a set of numbers...
I work on the principle that if something can be expressed numerically, that's always better than expressing it with code.
PATCH v2 5/27 arm: pci: add a align_resource hook
Russell,
Quoted 5 lines · Russell King - ARM Linux
Yep, otherwise we'll have yet more code to review rather than one algorithm with a set of numbers...
I work on the principle that if something can be expressed numerically, that's always better than expressing it with code.
Having a hook allows for more flexibility (for example having special alignment requirements depending on the device or something like that), but I don't need this flexibility for the specific case I'm interested in. So as you suggest, let's not over-engineer this, and use numerical values. I'll rework my patch according to this suggestion.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniQuoted 3 lines · Thomas Petazzoni
Quoted 2 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
BTW, this, as a general statement, is wrong - though it really depends what you mean by "region".
Remember that BARs can be set where-ever provided that they satisify their _individual_ alignment requirements. So, an IO bar which occupies 16 bytes must be set to a 16-byte boundary.
Now, there's an additional complication there which occurs if you have ISA devices sharing the PCI IO space: ISA devices used to only decode 10 bits of IO space, which means that their registers repeat throughout the IO space.
Therefore, it is generally accepted that within any 1K block, only the first 256 locations are only usable.
Moreover, some PCI cards have taken advantage of this, particularly VGA cards. For example, S3 VGA cards put different registers on 1K multiples of the standard PC VGA IO addresses...
Also, another reason why I suspect your statement is wrong if I were to interpret "region" as "BAR" is that consider a bunch of PCI peripherals behind a PCI bridge. The total number of IO BARs on the peripherals is 16.
If you allocate each of those IO BARs to be 4K aligned, then you consume all 64K of IO space behind one bridge, which leaves no space for any other IO peripherals elsewhere in the bus structure.
Last reason I think that intepretation is wrong is, on this PC, I see:
Region 4: I/O ports at 1800 [size=8]
Region 0: I/O ports at 1830 [size=8]
Region 2: I/O ports at 1840 [size=32]
Region 4: I/O ports at 1860 [size=32]
Region 4: I/O ports at 1880 [size=32]
Region 4: I/O ports at 18a0 [size=32]
Region 4: I/O ports at 18c0 [size=32]
Region 4: I/O ports at 18e0 [size=32]
Region 4: I/O ports at 1c00 [size=32]
Region 0: I/O ports at 1c48 [size=8]
Region 1: I/O ports at 183c [size=4]
Region 2: I/O ports at 1c40 [size=8]
Region 3: I/O ports at 1838 [size=4]
Region 4: I/O ports at 1c20 [size=32]
Region 4: I/O ports at 1c60 [size=32]which doesn't follow - and I can pull out other examples on other x86 platforms where IO BARs aren't aligned to 4K...
PATCH v2 5/27 arm: pci: add a align_resource hook
Russell,
Quoted 7 lines · Russell King - ARM Linux
Quoted 3 lines · Thomas Petazzoni
Quoted 2 lines · Thomas Petazzoni
The PCI specifications says that an I/O region must be aligned on a 4 KB boundary, and a memory region aligned on a 1 MB boundary.
BTW, this, as a general statement, is wrong - though it really depends what you mean by "region".
Yes, sorry, my statement does not correctly reflect the reality. My knowledge of the PCI terminology is still quite fuzzy (as you found out). What I am referring to is that the PCI standard requires the I/O base register of a PCI-to-PCI bridge to contain a 4 KB aligned address.
Quoted 1 line
From the PCI-to-PCI Bridge Architecture Specification, Revision 1.1,
section 3.2.5.6. I/O Base Register and I/O Limit Register:
"""
If a bridge implements an I/O address range, the upper 4 bits of both the I/O Base and I/O Limit registers are writable and correspond to address bits AD[15::12]. For the purpose of address decoding, the bridge assumes that the lower 12 address bits, AD[11::00], of the I/O base address (not implemented in the I/O Base register) are zero. Similarly, the bridge assumes that the lower 12 address bits, AD[11::00], of the I/O limit address (not implemented in the I/O Limit register) are FFFh. Thus, the bottom of the defined I/O address range will be aligned to a 4 KB boundary and the top of the defined I/O address range will be one less than a 4 KB boundary.
"""
And the Linux PCI resource allocation code complies with this, so that if I have two PCI-to-PCI bridges (each having downstream a device with an I/O BAR), then the first PCI-to-PCI bridge gets its I/O base address register set to ADDR + 0x0, and the second bridge gets its I/O base address set to ADDR + 0x1000. And this doesn't play well with the requirements of Marvell address decoding windows for PCIe I/O regions, which must be 64 KB aligned.
So I guess I should simply rewrite the commit log to make it clear that I'm referring to the I/O base address register of PCI-to-PCI bridges.
Would this be more correct? In that case, maybe in fact I really need a hook so that this alignment requirement on only applied on the resources allocated to bridges, and not on their downstream devices?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 7 lines · Thomas Petazzoni
And the Linux PCI resource allocation code complies with this, so that if I have two PCI-to-PCI bridges (each having downstream a device with an I/O BAR), then the first PCI-to-PCI bridge gets its I/O base address register set to ADDR +
0x0, and the second bridge gets its I/O base address set to ADDR +0x1000. And this doesn't play well with the requirements of Marvell address decoding windows for PCIe I/O regions, which must be 64 KB aligned.
But we normally only assign a 64 KB I/O window to each PCI host bridge. Requiring PCI bridges to be space 64 KB apart would mean that we cannot actually support bridges at all.
Is this just about your "virtual" bridges? If each one has its own 64 KB I/O range and its own configuration space, that sounds a lot like you should make them appear as individual domains instead.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 9 lines · Arnd Bergmann
Quoted 7 lines · Thomas Petazzoni
And the Linux PCI resource allocation code complies with this, so that if I have two PCI-to-PCI bridges (each having downstream a device with an I/O BAR), then the first PCI-to-PCI bridge gets its I/O base address register set to ADDR +
0x0, and the second bridge gets its I/O base address set to ADDR +0x1000. And this doesn't play well with the requirements of Marvell address decoding windows for PCIe I/O regions, which must be 64 KB aligned.
Quoted 7 lines
But we normally only assign a 64 KB I/O window to each PCI host bridge. Requiring PCI bridges to be space 64 KB apart would mean that we cannot actually support bridges at all.
Is this just about your "virtual" bridges? If each one has its own 64 KB I/O range and its own configuration space, that sounds a lot like you should make them appear as individual domains instead.
Yes, it is about the emulated PCI-to-PCI bridges. Each emulated PCI-to-PCI bridge corresponds to one hardware PCIe interface, and I need the I/O base address assigned to each PCIe interface to be aligned on a 64 KB boundary. I am not sure to understand why you think this is a problem.
Also, what do you mean exactly by making them appear as individual domains?
Remember that the very reason to use emulated PCI-to-PCI bridges is that we want to assign a global range of addresses of I/O regions and a global range of addresses of memory regions, and let the Linux PCI core allocate from those two ranges to the different devices connected downstream of the PCI-to-PCI bridges. This gives us for free the rather complex allocation of addresses we need to set up our address decoding windows.
If we have have separate domains for each of our hardware PCIe interface, can we still benefit from this allocation of resources from a globally defined range of I/O addresses and memory addresses?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 8 lines · Thomas Petazzoni
Yes, it is about the emulated PCI-to-PCI bridges. Each emulated PCI-to-PCI bridge corresponds to one hardware PCIe interface, and I need the I/O base address assigned to each PCIe interface to be aligned on a 64 KB boundary. I am not sure to understand why you think this is a problem.
Also, what do you mean exactly by making them appear as individual domains?
I mean you could make each root port look like a separate host bridge that is not related to the others, and not have any emulated PCI-to-PCI bridges at all.
Quoted 11 lines
Remember that the very reason to use emulated PCI-to-PCI bridges is that we want to assign a global range of addresses of I/O regions and a global range of addresses of memory regions, and let the Linux PCI core allocate from those two ranges to the different devices connected downstream of the PCI-to-PCI bridges. This gives us for free the rather complex allocation of addresses we need to set up our address decoding windows.
If we have have separate domains for each of our hardware PCIe interface, can we still benefit from this allocation of resources from a globally defined range of I/O addresses and memory addresses?
My interpretation of what you told me in the previous mail is that each root port has
* A separate configuration space
* A separate 64KB I/O window that is not shared with the other ports, or potentially multiple 64KB windows, which we would not want to use
* A configurable range of the memory space that does not overlap with the other ports
Is the above a correct description?
If so, I think it would be most sensible to not try to put all ports into the same domain, but give each port the full view of its own 256 buses, and 64KB I/O space. The memory space can still be directly mapped, if you only set up the physical address window for that after the bus scan is complete.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 3 lines · Arnd Bergmann
I mean you could make each root port look like a separate host bridge that is not related to the others, and not have any emulated PCI-to-PCI bridges at all.
Ok.
Quoted 21 lines
Quoted 11 lines
Remember that the very reason to use emulated PCI-to-PCI bridges is that we want to assign a global range of addresses of I/O regions and a global range of addresses of memory regions, and let the Linux PCI core allocate from those two ranges to the different devices connected downstream of the PCI-to-PCI bridges. This gives us for free the rather complex allocation of addresses we need to set up our address decoding windows.
If we have have separate domains for each of our hardware PCIe interface, can we still benefit from this allocation of resources from a globally defined range of I/O addresses and memory addresses?
My interpretation of what you told me in the previous mail is that each root port has
* A separate configuration space
* A separate 64KB I/O window that is not shared with the other ports, or potentially multiple 64KB windows, which we would not want to use
* A configurable range of the memory space that does not overlap with the other ports
Quoted 7 lines
Is the above a correct description?
If so, I think it would be most sensible to not try to put all ports into the same domain, but give each port the full view of its own 256 buses, and 64KB I/O space. The memory space can still be directly mapped, if you only set up the physical address window for that after the bus scan is complete.
Does this still allows me to give the Linux PCI *one* global range of addresses for I/O space, and *one* global range of addresses for memory space, and the the Linux PCI core assign ranges, within those global ranges, to each host bridge?
This is absolutely essential for me, as I then read those allocated ranges to configure the address decoding windows.
Basically, I have currently two suggestions:
* From Jason Gunthorpe, to not use any host bridge, and instead use only PCI-to-PCI bridges, one per PCIe interface.
* From you, to not use any PCI-to-PCI bridge, and use only host bridges, one per PCIe interface.
Would it be possible to get some consensus on this? In the review of RFCv1, I was already told to use one global host bridge, and then one PCI-to-PCI bridge per PCIe interface, and now we're talking about doing something different. I'd like to avoid having to try gazillions of different possible implementations :-)
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Arnd,
Quoted 7 lines · Thomas Petazzoni
Basically, I have currently two suggestions:
* From Jason Gunthorpe, to not use any host bridge, and instead use only PCI-to-PCI bridges, one per PCIe interface.
* From you, to not use any PCI-to-PCI bridge, and use only host bridges, one per PCIe interface.
Thinking more about this, this solution (using one emulated host bridge per PCIe interface) would cause one problem: the PCIe device itself would no longer be in slot 0.
If I'm correct, with one host bridge per PCIe interface, we would have the following topology:
bus 0, slot 0: emulated host bridge 0
bus 0, slot 1: PCIe device connected to PCIe interface 0 bus 1, slot 0: emulated host bridge 1
bus 1, slot 1: PCIe device connected to PCIe interface 1 bus 2, slot 0: emulated host bridge 2
bus 2, slot 1: PCIe device connected to PCIE interface 2 etc.
However, one of the reason to use a PCI-to-PCI bridge was to ensure that the PCIe devices were all listed in slot 0. According to the Marvell engineers who work on the PCIe stuff, some new PCIe devices have this requirement. I don't have a lot of details about this, but I was told that most of the new Intel NICs require this, for example the Intel X520 fiber NIC. Maybe PCIe experts (Jason?) could provide more details about this, and confirm/infirm this statement.
The usage of PCI-to-PCI bridge allows to have each PCIe device on its own bus, at slot 0, which also solves this problem.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 9 lines · Thomas Petazzoni
Arnd,
Quoted 5 lines · Thomas Petazzoni
Basically, I have currently two suggestions:
* From Jason Gunthorpe, to not use any host bridge, and instead use only PCI-to-PCI bridges, one per PCIe interface.
Quoted 6 lines
Quoted 2 lines
* From you, to not use any PCI-to-PCI bridge, and use only host bridges, one per PCIe interface.
Thinking more about this, this solution (using one emulated host bridge per PCIe interface) would cause one problem: the PCIe device itself would no longer be in slot 0.
I think that's device 0 not slot 0 right?
Quoted 5 lines
If I'm correct, with one host bridge per PCIe interface, we would have the following topology:
bus 0, slot 0: emulated host bridge 0
bus 0, slot 1: PCIe device connected to PCIe interface 0
I /think/ the bus that the root port itself is on is different from the bus that the downstream device is on, so wouldn't you end up with:
bus 0, slot 0: emulated host bridge 0
bus 1, slot 0: PCIe device connected to PCIe interface 0
(and isn't that "root port" not "host bridge" in the first line above?)
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniQuoted 11 lines · Thomas Petazzoni
Arnd,
Quoted 7 lines · Thomas Petazzoni
Basically, I have currently two suggestions:
* From Jason Gunthorpe, to not use any host bridge, and instead use only PCI-to-PCI bridges, one per PCIe interface.
* From you, to not use any PCI-to-PCI bridge, and use only host bridges, one per PCIe interface.
Arnd is suggesting to use multiple *linux* host bridges (ie host drivers), there is never any need for a 'host bridge config space' as in patch #7, in either case.
Quoted 7 lines
However, one of the reason to use a PCI-to-PCI bridge was to ensure that the PCIe devices were all listed in slot 0. According to the Marvell engineers who work on the PCIe stuff, some new PCIe devices have this requirement. I don't have a lot of details about this, but I was told that most of the new Intel NICs require this, for example the Intel X520 fiber NIC. Maybe PCIe experts (Jason?) could provide more details about this, and confirm/infirm this statement.
I'm not sure what this is referring to.. I don't recall any specific requirements in PCI-E for the device number, I think the spec requires it to be learned based on the config TLPs received.
There might be a device number sensitivity in INTx translation, but that is defined by the spec.
That said, if your root complex is PCI-E compliant then all downstream end ports attached to a root port should have a device number of 0.
Quoted 2 lines
The usage of PCI-to-PCI bridge allows to have each PCIe device on its own bus, at slot 0, which also solves this problem.
Hrm....
Looking at the docs, you will also need to change the internal device number (probably reg 41a04 again) to something other than 0, otherwise the Marvell itself will claim device number 0 and the downstream end port will be device number 1. You should see this happen today??
You should set the Marvell internal device number to something like all ones and then deny any Linux config register access to the all ones device number on the subordinate bus to hide the Marvell end port config space registers from Linux.
As for as this process goes, it doesn't matter which approach you take. If you use multiple PCI domains then you'd still be able to arrange things via the above so that the downstream device could always claim device number 0.
Jason
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniQuoted 13 lines · Thomas Petazzoni
Does this still allows me to give the Linux PCI one global range of addresses for I/O space, and one global range of addresses for memory space, and the the Linux PCI core assign ranges, within those global ranges, to each host bridge?
This is absolutely essential for me, as I then read those allocated ranges to configure the address decoding windows.
Basically, I have currently two suggestions:
* From Jason Gunthorpe, to not use any host bridge, and instead use only PCI-to-PCI bridges, one per PCIe interface.
Quoted 8 lines
* From you, to not use any PCI-to-PCI bridge, and use only host bridges, one per PCIe interface.
Would it be possible to get some consensus on this? In the review of RFCv1, I was already told to use one global host bridge, and then one PCI-to-PCI bridge per PCIe interface, and now we're talking about doing something different. I'd like to avoid having to try gazillions of different possible implementations :-)
I'm actually fine with either of the two suggestions you mentioned above, whichever is easier to implement and/or more closely matches what the hardware actually implements is better IMHO.
The part that I did not like about having emulated PCI-to-PCI bridges is that it seems to just work around a (percieved or real) limitation in the Linux kernel by adding a piece of infrastructure, rather than lifting that limitation by making the kernel deal with what the hardware provides. That reminded me of the original mach-vt8500 PCI implementation that faked a complete PCI host bridge and a bunch of PCI devices on it, in order to use the via-velocity ethernet controller, instead of adding a simple 'platform_driver' struct to that driver.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 9 lines · Arnd Bergmann
I'm actually fine with either of the two suggestions you mentioned above, whichever is easier to implement and/or more closely matches what the hardware actually implements is better IMHO.
The part that I did not like about having emulated PCI-to-PCI bridges is that it seems to just work around a (percieved or real) limitation in the Linux kernel by adding a piece of infrastructure, rather than lifting that limitation by making the kernel deal with what the hardware provides. That reminded me of the original mach-vt8500
Well.. in this case there is a standard - PCI-E for what HW vendors are supposed to do. The kernel core code follows it and works with compliant hardware.
Marvell HW is not compliant.
So..
Should the kernel core PCI code support this particular non-compliance? Should the driver work around the non-compliance and present a compliant interface to the kernel and userspace?
My take is the kernel core PCI code is fine, and I hope this will be an isolated issue with one family of Marvell IP. So working around the HW problem in the driver seems best.
If we learn of many more instances like this then, yah, update the core code and rip out this driver work around...
Jason
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 16 lines · Jason Gunthorpe
Quoted 5 lines · Arnd Bergmann
The part that I did not like about having emulated PCI-to-PCI bridges is that it seems to just work around a (percieved or real) limitation in the Linux kernel by adding a piece of infrastructure, rather than lifting that limitation by making the kernel deal with what the hardware provides. That reminded me of the original mach-vt8500
Well.. in this case there is a standard - PCI-E for what HW vendors are supposed to do. The kernel core code follows it and works with compliant hardware.
Marvell HW is not compliant.
So..
Quoted 7 lines
Should the kernel core PCI code support this particular non-compliance? Should the driver work around the non-compliance and present a compliant interface to the kernel and userspace?
My take is the kernel core PCI code is fine, and I hope this will be an isolated issue with one family of Marvell IP. So working around the HW problem in the driver seems best.
I don't remember the kernel ever caring about whether hardware complies to a standard or not. The kernel's job is to make hardware work, based on the actual implementation of that hardware. In a lot of cases that means taking the standard document as a reference, and adding quirks for the devices that are different.
In the end, it comes down to the impact on the code complexity, and the run-time overhead for whatever hardware is most common when adding those quirks.
Can you (or someone else) describe what kind of changes to the core code we would actually need to make it work without emulting the bridge?
Quoted 2 lines
If we learn of many more instances like this then, yah, update the core code and rip out this driver work around...
But the code was specifically written to be reusable, which is normally a good thing.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 9 lines · Arnd Bergmann
I don't remember the kernel ever caring about whether hardware complies to a standard or not. The kernel's job is to make hardware work, based on the actual implementation of that hardware. In a lot of cases that means taking the standard document as a reference, and adding quirks for the devices that are different.
In the end, it comes down to the impact on the code complexity, and the run-time overhead for whatever hardware is most common when adding those quirks.
This is not only about standards, it is also about re-using the PCI resource allocation code.
In my RFCv1, sent December, 7th, I wasn't using any emulated PCI-to-PCI bridge. So it *can* perfectly work without it.
However, one major drawback of my RFCv1 version is that since I didn't know how much I/O space and memory space was needed for each PCIe device, I had to oversize the address decoding windows. And also, I had to have a special allocator (certainly simple, but still) to find an available physical address to set up each address decoding window.
Emulating a PCI-to-PCI bridge very nicely allows to re-use the PCI core resource allocation code. I think it's really the main reason for emulated those PCI-to-PCI bridges, rather than willing to comply to some standards.
So what I'm going to do now is rework my patch series by removing the emulated host bridge (which is normally mandatory by PCIe standard, but Linux doesn't need it, so we don't care), but I'll keep the emulated PCI-to-PCI bridges in order to benefit for the PCI core resource allocation mechanisms.
Is this ok for you?
I'd like to settle on the strategy to follow, because we're really going a funny road here: on December 7th, I submit a series that doesn't use any PCI-to-PCI bridge, and I'm told that I should emulate some. I spent a long time working on an implementation that uses emumlated PCI-to-PCI bridges, which I submitted on Monday, now to be told that I should work really hard not to use PCI-to-PCI bridges. I hope you can feel my little embarrassment here...
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 14 lines · Thomas Petazzoni
Dear Arnd Bergmann,
Quoted 9 lines · Arnd Bergmann
I don't remember the kernel ever caring about whether hardware complies to a standard or not. The kernel's job is to make hardware work, based on the actual implementation of that hardware. In a lot of cases that means taking the standard document as a reference, and adding quirks for the devices that are different.
In the end, it comes down to the impact on the code complexity, and the run-time overhead for whatever hardware is most common when adding those quirks.
Quoted 5 lines
This is not only about standards, it is also about re-using the PCI resource allocation code.
In my RFCv1, sent December, 7th, I wasn't using any emulated PCI-to-PCI bridge. So it *can* perfectly work without it.
Ok, Isee.
Quoted 5 lines
However, one major drawback of my RFCv1 version is that since I didn't know how much I/O space and memory space was needed for each PCIe device, I had to oversize the address decoding windows. And also, I had to have a special allocator (certainly simple, but still) to find an available physical address to set up each address decoding window.
Well, for the I/O space, there is no oversizing because either way you end up with exactly 64KB per root port, right?
Quoted 5 lines
Emulating a PCI-to-PCI bridge very nicely allows to re-use the PCI core resource allocation code. I think it's really the main reason for emulated those PCI-to-PCI bridges, rather than willing to comply to some standards.
Quoted 7 lines
So what I'm going to do now is rework my patch series by removing the emulated host bridge (which is normally mandatory by PCIe standard, but Linux doesn't need it, so we don't care), but I'll keep the emulated PCI-to-PCI bridges in order to benefit for the PCI core resource allocation mechanisms.
Is this ok for you?
Using the Linux allocator for memory resources does sound useful, so if that requires using the emulated PCI-to-PCI bridges, I guess it's the best compromise.
Quoted 7 lines
I'd like to settle on the strategy to follow, because we're really going a funny road here: on December 7th, I submit a series that doesn't use any PCI-to-PCI bridge, and I'm told that I should emulate some. I spent a long time working on an implementation that uses emumlated PCI-to-PCI bridges, which I submitted on Monday, now to be told that I should work really hard not to use PCI-to-PCI bridges. I hope you can feel my little embarrassment here...
I'm sorry about this situation. Unfortunatly the way that such decisions are made is not always straightforward, and what seems like a good idea at one point turns out to be a mistake or more complex than anticipated later. With the description of the first patch set, I did not think it would be necessary to fake a bridge device and their config space.
What I had guessed you could do was to call pci_scan_root_bus on each of the ports, and then set up the memory space window for the bus including all of its child devices.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 8 lines · Arnd Bergmann
Quoted 5 lines
However, one major drawback of my RFCv1 version is that since I didn't know how much I/O space and memory space was needed for each PCIe device, I had to oversize the address decoding windows. And also, I had to have a special allocator (certainly simple, but still) to find an available physical address to set up each address decoding window.
Well, for the I/O space, there is no oversizing because either way you end up with exactly 64KB per root port, right?
Correct I/O space is not an issue, of course. Only the memory windows are an issue (in terms of quantity of address space used).
That said, the PCI-to-PCI bridge solution doesn't solve the fact that I/O addresses get assigned even though the driver will most likely use them. This means that one I/O window is consumed for each PCIe interface, even though it is not being used in practice. And see I have 10 PCIe interfaces in this SoC, and only 20 windows available globally (not only for PCIe, but also for NAND, NOR, etc.). But for now, I'd like to leave this potential problem on the side, and get something working. If it seems useful to remove this problem later, we'll work on it.
Quoted 11 lines
Quoted 11 lines
Emulating a PCI-to-PCI bridge very nicely allows to re-use the PCI core resource allocation code. I think it's really the main reason for emulated those PCI-to-PCI bridges, rather than willing to comply to some standards.
So what I'm going to do now is rework my patch series by removing the emulated host bridge (which is normally mandatory by PCIe standard, but Linux doesn't need it, so we don't care), but I'll keep the emulated PCI-to-PCI bridges in order to benefit for the PCI core resource allocation mechanisms.
Quoted 5 lines
Quoted 1 line
Is this ok for you?
Using the Linux allocator for memory resources does sound useful, so if that requires using the emulated PCI-to-PCI bridges, I guess it's the best compromise.
Yes, that was Jason's original idea when he suggested to use PCI-to-PCI bridges. And when I did the implementation, it really worked nicely. And this PCI-to-PCI bridge emulation stuff is really not a big deal, look at drivers/pci/sw-pci-pci-bridge.c: 185 lines in total, including 10 lines of comment header at the top.
Quoted 5 lines
I'm sorry about this situation. Unfortunatly the way that such decisions are made is not always straightforward, and what seems like a good idea at one point turns out to be a mistake or more complex than anticipated later. With the description of the first patch set, I did not think it would be necessary to fake a bridge device and their config space.
Sure, I understand this. I guess you also understand my slight frustration when I propose A, I'm told to do B, I propose B, and I'm then suggested to do A again :-) But I agree, it's part of the technical discussion, and we can't get it right on the first shot.
Quoted 3 lines
What I had guessed you could do was to call
pci_scan_root_buson each of the ports, and then set up the memory space window for the bus including all of its child devices.
But where would I read how much space is needed for the I/O and memory regions of each bus?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Thomas PetazzoniQuoted 3 lines · Thomas Petazzoni
So what I'm going to do now is rework my patch series by removing the emulated host bridge (which is normally mandatory by PCIe standard, but Linux doesn't need it, so we don't care), ...
This is a tangent since you're removing the emulated host bridge anyway, but it's been mentioned a couple of times, and I'd like to understand this. Jason mentioned earlier in the [07/27] emulated host bridge thread that the PCIe spec requires a host bridge at 00:00.0. I've never seen that mentioned in the spec; can somebody point me to the actual requirement that host bridges appear in config space?
My understanding has been that host bridges, whether PCI or PCIe, are required to *exist*, but that the way you enumerate them and configure them is outside the scope of the PCI/PCIe specs. I know that many chips, especially for x86, *do* make the host bridge appear in config space, but I've never seen a requirement for that.
Bjorn
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 6 lines · Bjorn Helgaas
This is a tangent since you're removing the emulated host bridge anyway, but it's been mentioned a couple of times, and I'd like to understand this. Jason mentioned earlier in the [07/27] emulated host bridge thread that the PCIe spec requires a host bridge at 00:00.0. I've never seen that mentioned in the spec; can somebody point me to the actual requirement that host bridges appear in config space?
Hum, a more careful search/reading brings up this:
7.2.2.1. Host Bridge Requirements
[...] The use of Host Bridge PCI class code is reserved for backwards compatibility; host Bridge configuration space is opaque to standard PCI Express software and may be implemented in an implementation specific manner that is compatible with PCI Host Bridge Type 0 configuration space. A PCI Express Host Bridge is not required to signal errors through a Root Complex Event Collector. This support is optional for PCI Express Host Bridges.
So, if it is present it is required to be compatible with the 'PCI Host Bridge' stuff, but it is not mandatory.
My bad, I believe I got also confused with the spec language regarding a 'host bridge' vs a 'host bridge configuration space'
Regards,
Jason
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Arnd BergmannQuoted 9 lines · Arnd Bergmann
Quoted 7 lines · Thomas Petazzoni
And the Linux PCI resource allocation code complies with this, so that if I have two PCI-to-PCI bridges (each having downstream a device with an I/O BAR), then the first PCI-to-PCI bridge gets its I/O base address register set to ADDR +
0x0, and the second bridge gets its I/O base address set to ADDR +0x1000. And this doesn't play well with the requirements of Marvell address decoding windows for PCIe I/O regions, which must be 64 KB aligned.
Quoted 3 lines
But we normally only assign a 64 KB I/O window to each PCI host bridge. Requiring PCI bridges to be space 64 KB apart would mean that we cannot actually support bridges at all.
The PCI resource code uses full 32 bit integers when it handles IO addresses, so this actually does sort of work out.
However, Thomas how did you recover the high bits of the IO window address from the bridge configuration? Are you reading the struct resource directly? That probably causes problems with hotplug/etc...
If you look back in your old emails I outlined a solution to this using the MMU.
Jason
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Jason Gunthorpe,
Quoted 4 lines · Jason Gunthorpe
However, Thomas how did you recover the high bits of the IO window address from the bridge configuration? Are you reading the struct resource directly? That probably causes problems with hotplug/etc...
The PCI-to-PCI bridge configuration space has a register with the high bits of the I/O window address. If you look at the PCI-to-PCI emulation code, I set the bit that says "I'm a bridge capable of 32 bits addressing of I/O addresses", and then when setting up the windows, I reconstruct the full 32 bits address by reading the two I/O address registers.
See 3.2.5.6 in the PCI-to-PCI bridge specification:
If the low four bits of the I/O Base and I/O Limit registers are 01h, then the bridge supports 32-bit I/O address decoding, and the I/O Base Upper 16 Bits and the I/O Limit Upper 16 Bits hold the upper 16 bits, corresponding to AD[31::16], of the 32-bit I/O Base and I/O Limit addresses respectively. In this case, system configuration software is permitted to locate the I/O address range supported by the anywhere in the 4-GB I/O Space. Note that the 4-KB alignment and granularity restrictions still apply when the bridge supports 32 -bit I/O addressing.
(And my code does ensure that the low four bits of the I/O Base and I/O Limit registers are 01h)
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
In reply to Jason GunthorpeQuoted 6 lines · Jason Gunthorpe
Quoted 3 lines
But we normally only assign a 64 KB I/O window to each PCI host bridge. Requiring PCI bridges to be space 64 KB apart would mean that we cannot actually support bridges at all.
The PCI resource code uses full 32 bit integers when it handles IO addresses, so this actually does sort of work out.
However, we only reserve 1 MB (I think) virtual address window for all I/O spaces of all PCI domains combined, at a fixed location (0xfee00000). This means we can have at most 16 such windows at run-time. That can be changed if necessary, but it seems like overkill when in practice you only need a few bytes at most.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · Jason Gunthorpe
Quoted 3 lines
But we normally only assign a 64 KB I/O window to each PCI host bridge. Requiring PCI bridges to be space 64 KB apart would mean that we cannot actually support bridges at all.
Quoted 8 lines
Quoted 2 lines
The PCI resource code uses full 32 bit integers when it handles IO addresses, so this actually does sort of work out.
However, we only reserve 1 MB (I think) virtual address window for all I/O spaces of all PCI domains combined, at a fixed location (
0xfee00000). This means we can have at most 16 such windows at run-time. That can be changed if necessary, but it seems like overkill when in practice you only need a few bytes at most.
I am not sure where this 0xfee00000 address comes from, but in my case (and I think in the Tegra PCI driver as well), we tell the Linux PCI core from which addresses the I/O ranges should be allocated. In my DT, I have:
ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */
0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */
0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */
0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */
0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */
0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */
0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */
0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */
0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */
0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */
0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */And then, the Marvell PCI driver gets the "downstream I/O" range, parses it into a "struct resource", and then does (where &pcie->io is the struct resource into which we parsed the "downstream I/O" range):
pci_add_resource_offset(&sys->resources, &pcie->io, sys->io_offset); [...] pci_ioremap_io(nr * SZ_64K, pcie->io.start);
And it works just fine, I get my I/O ranges allocated at 0xc0000000 for the first device, 0xc0010000 (i.e base address + 64KB) for the second device, etc.
The Tegra PCI driver does exactly the same (I shamelessly copied what Thierry has done).
I somehow have the feeling that we are looking for problems that simply don't exist...
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 5/27 arm: pci: add a align_resource hook
Quoted 18 lines · Thomas Petazzoni
I am not sure where this
0xfee00000address comes from, but in my case (and I think in the Tegra PCI driver as well), we tell the Linux PCI core from which addresses the I/O ranges should be allocated. In my DT, I have:ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ 0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ 0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */ 0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */ 0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */ 0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */ 0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */
Quoted 7 lines
And then, the Marvell PCI driver gets the "downstream I/O" range, parses it into a "struct resource", and then does (where
&pcie->iois the struct resource into which we parsed the "downstream I/O" range):pci_add_resource_offset(&sys->resources, &pcie->io, sys->io_offset); [...] pci_ioremap_io(nr * SZ_64K, pcie->io.start);
0xfee00000 is the platform independent virtual address that pci_ioremap_io maps your platform specific physical address (from pcie->io.start) to. It's defined (in the kernel I am looking at) in asm/io.h as
#define PCI_IO_VIRT_BASE 0xfee00000
and used by pci_ioremap_io as
return ioremap_page_range(PCI_IO_VIRT_BASE + offset,
PCI_IO_VIRT_BASE + offset + SZ_64K,
phys_addr,
__pgprot(get_mem_type(MT_DEVICE)->prot_pte));Quoted 3 lines
And it works just fine, I get my I/O ranges allocated at
0xc0000000for the first device,0xc0010000(i.e base address + 64KB) for the second device, etc.
(void*)0xc0000000 is the normal PAGE_OFFSET. If you map your I/O space there, you are in big trouble because that is supposed to have the start of your physical memory mapping.
Arnd
PATCH v2 5/27 arm: pci: add a align_resource hook
Dear Arnd Bergmann,
Quoted 14 lines · Arnd Bergmann
0xfee00000is the platform independent virtual address thatpci_ioremap_iomaps your platform specific physical address (frompcie->io.start) to. It's defined (in the kernel I am looking at) inasm/io.has#define
PCI_IO_VIRT_BASE0xfee00000and used by
pci_ioremap_ioasreturn ioremap_page_range(PCI_IO_VIRT_BASE + offset, PCI_IO_VIRT_BASE + offset + SZ_64K, phys_addr, __pgprot(get_mem_type(MT_DEVICE)->prot_pte));
Quoted 7 lines
Quoted 3 lines
And it works just fine, I get my I/O ranges allocated at
0xc0000000for the first device,0xc0010000(i.e base address + 64KB) for the second device, etc.(void*)0xc0000000 is the normal
PAGE_OFFSET. If you map your I/O space there, you are in big trouble because that is supposed to have the start of your physical memory mapping.
Aaah, I know where the confusion comes from. You are talking about virtual addresses, while I am talking about physical addresses. 0xC0000000 in my DT is a *physical* address.
Basically, with Marvell SoCs, we have the following behavior:
------- --------------------
Virtual address --> | MMU | --> Physical address --> | Address decoding | --> real hardware
------- --------------------The MMU is the usual stuff everybody knows about. What's more special about Marvell SoC is this "Address decoding" thing. Basically, instead of having the physical address of things fully hardcoded and mentioned in the datasheets, they are configurable. So for each PCIe interface, you have to set up an address decoding window for the I/O accesses and another address decoding window for the memory accesses. And the physical address associated to each of these "address decoding windows" can be freely chosen, so they must be "assigned" for each PCIe interface.
So, my 0xC0000000 is a *physical* address is the diagram above. The fact that it gets maps at 0xfee00000 as a virtual address doesn't really matter for me, I'm just fine with that.
Does that clarify things?
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 6/27 lib: devres: don't enclose pcim_*() functions in CONFIG_HAS_IOPORT
In reply to Thomas PetazzoniThe pcim_*() functions are used by the libata-sff subsystem, and this subsystem is used for many SATA drivers on ARM platforms that do not necessarily have I/O ports.
--- lib/devres.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/devres.c +1 -1
@@ -195,6 +195,7 @@ void devm_ioport_unmap(struct device *dev, void __iomem *addr) devm_ioport_map_match, (void *)addr)); }EXPORT_SYMBOL (devm_ioport_unmap); +#endif /* CONFIG_HAS_IOPORT */ #ifdef CONFIG_PCI /*
@@ -400,4 +401,3 @@ void pcim_iounmap_regions(struct pci_dev *pdev, int mask) }EXPORT_SYMBOL (pcim_iounmap_regions);#endif /* CONFIG_PCI */ -#endif /* CONFIG_HAS_IOPORT */
Signature · 1 line
1.7.9.5
PATCH v2 7/27 PCI: Add software-emulated host bridge
In reply to Thomas PetazzoniFrom: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]
--- drivers/pci/Kconfig | 3 + drivers/pci/Makefile | 3 + drivers/pci/sw-host-bridge.c | 144 ++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 23 +++++++ 4 files changed, 173 insertions(+) create mode 100644 drivers/pci/sw-host-bridge.c
drivers/pci/Kconfig +3 -0
@@ -119,3 +119,6 @@ config PCI_IOAPIC config PCI_LABEL def_bool y if (DMI || ACPI) select NLS + +config PCI_SW_HOST_BRIDGE + bool
drivers/pci/Makefile +3 -0
@@ -15,6 +15,9 @@ obj-$(CONFIG_PCIEPORTBUS) += pcie/ obj-$(CONFIG_PCI_IOAPIC) += ioapic.o +# Emulated PCI elements +obj-$(CONFIG_PCI_SW_HOST_BRIDGE) += sw-host-bridge.o + # Build the PCI Hotplug drivers if we were asked to obj-$(CONFIG_HOTPLUG_PCI) += hotplug/ ifdef CONFIG_HOTPLUG_PCI
drivers/pci/sw-host-bridge.c new file +144 -0
144 lines not on this page · raw message
include/linux/pci.h +23 -0
@@ -1841,4 +1841,27 @@ static inline struct eeh_dev *pci_dev_to_eeh_dev(struct pci_dev *pdev)*/ struct pci_dev *pci_find_upstream_pcie_bridge (struct pci_dev *pdev); +struct pci_sw_host_bridge { +u16 vendor; +u16 device; +u16 command; +u16 status; +u16 class; +u8 interface; +u8 revision; +u8 bist; +u8 header_type; +u8 latency_timer; +u8 cache_line_size; +u32 bar[6 ]; +u16 subsystem_vendor; +u16 subsystem_device; +}; + +extern int pci_sw_host_bridge_init (struct pci_sw_host_bridge *bridge); +extern int pci_sw_host_bridge_read (struct pci_sw_host_bridge *bridge, +unsigned int where,int size,u32 *value); +extern int pci_sw_host_bridge_write (struct pci_sw_host_bridge *bridge, +unsigned int where,int size,u32 value); +#endif /* LINUX_PCI_H */
Signature · 1 line
1.7.9.5
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 9 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]Signed-off-by Thierry Reding <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>
Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
Arnd
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 11 lines · Arnd Bergmann
Quoted 9 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]Signed-off-by Thierry Reding <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>
Quoted 6 lines
Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
At least on Tegra, there is no HW that exposes PCI configuration registers for the host bridge itself. Only the root ports have exposed PCI configuration registers. There was some debate re: whether a host bridge device needed to exist or not. This patch makes such a device exist if it's required.
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Arnd Bergmann
Quoted 9 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]Signed-off-by Thierry Reding <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>Quoted 6 lines
Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
Quoted 5 lines
At least on Tegra, there is no HW that exposes PCI configuration registers for the host bridge itself. Only the root ports have exposed PCI configuration registers. There was some debate re: whether a host bridge device needed to exist or not. This patch makes such a device exist if it's required.
If Linux will discover properly (I strongly suspect it does) without the host bridge, then I would say to ditch this...
The PCI-E standard requires a host bridge device, but if Linux doesn't require it then there is no reason to emulate one.
That would simplify the question of PCI IDs - for Marvell's case and the sw root port bridge we can just copy the IDs from the bogus config space of the HW.
Jason
PATCH v2 7/27 PCI: Add software-emulated host bridge
Dear Jason Gunthorpe,
Quoted 9 lines · Jason Gunthorpe
If Linux will discover properly (I strongly suspect it does) without the host bridge, then I would say to ditch this...
The PCI-E standard requires a host bridge device, but if Linux doesn't require it then there is no reason to emulate one.
That would simplify the question of PCI IDs - for Marvell's case and the sw root port bridge we can just copy the IDs from the bogus config space of the HW.
Not sure what you mean in this last paragraph. In this second version, I really rely on the emulated PCI-to-PCI bridges for the resource allocation. I give the Linux PCI core a global range of addresses for memory regions and a global range of addresses for I/O regions, and then I let Linux do the allocation of ranges on a per bridge basis, depending on the devices detected downstream. And at the end, I use those allocated ranges to set up the address decoding windows.
This all comes from your suggestions during the review of the first revision of this patch set.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 6 lines · Thomas Petazzoni
Quoted 3 lines
That would simplify the question of PCI IDs - for Marvell's case and the sw root port bridge we can just copy the IDs from the bogus config space of the HW.
Not sure what you mean in this last paragraph. In this second version, I really rely on the emulated PCI-to-PCI bridges for the resource
I'm refering to your earlier question about what PCI IDs to use for the SW emulated devices. If there is no need for the host bridge then you only need 1 PCI ID (for the root port bridge) and you can probably fairly safely re-use the one in the Marvell config space of the HW.
Quoted 5 lines
allocation. I give the Linux PCI core a global range of addresses for memory regions and a global range of addresses for I/O regions, and then I let Linux do the allocation of ranges on a per bridge basis, depending on the devices detected downstream. And at the end, I use those allocated ranges to set up the address decoding windows.
Yes, that all seems OK to me.
Jason
PATCH v2 7/27 PCI: Add software-emulated host bridge
Dear Jason Gunthorpe,
Quoted 4 lines · Jason Gunthorpe
I'm refering to your earlier question about what PCI IDs to use for the SW emulated devices. If there is no need for the host bridge then you only need 1 PCI ID (for the root port bridge) and you can probably fairly safely re-use the one in the Marvell config space of the HW.
Ah, ok, I see. But isn't a host bridge needed to bind all the PCI-to-PCI bridges under a single bus, in order to get the global resource assignment I was referring to?
Regarding the PCI IDs, I have started to work with Marvell to see what is possible. I, unfortunately, haven't received the answer for now.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 4 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
On Mon, 28 Jan 2013 15:23:48 -0700, Jason Gunthorpe wrote:
Quoted 8 lines
Quoted 4 lines
I'm refering to your earlier question about what PCI IDs to use for the SW emulated devices. If there is no need for the host bridge then you only need 1 PCI ID (for the root port bridge) and you can probably fairly safely re-use the one in the Marvell config space of the HW.
Ah, ok, I see. But isn't a host bridge needed to bind all the PCI-to-PCI bridges under a single bus, in order to get the global resource assignment I was referring to?
The PCI-E spec requires it, but AFAIK it doesn't actually *do* anything on Linux, and Linux doesn't require it.
I thought Thierry did this experiment and decided it wasn't necessary:
Quoted 7 lines
The reason is that with the latest bindings the matching of root ports to device tree nodes works as-is and nothing else indicates that the emulated host bridge is actually required to make any of this work. So in order not to introduce unneeded code I've left it out for now. If somebody decides that we actually need this host bridge (for standards compliance or whatnot) it could easily be added back.
Jason
PATCH v2 7/27 PCI: Add software-emulated host bridge
Dear Jason Gunthorpe,
Quoted 5 lines · Jason Gunthorpe
Quoted 5 lines
Quoted 4 lines
I'm refering to your earlier question about what PCI IDs to use for the SW emulated devices. If there is no need for the host bridge then you only need 1 PCI ID (for the root port bridge) and you can probably fairly safely re-use the one in the Marvell config space of the HW.
Quoted 8 lines
Quoted 3 lines
Ah, ok, I see. But isn't a host bridge needed to bind all the PCI-to-PCI bridges under a single bus, in order to get the global resource assignment I was referring to?
The PCI-E spec requires it, but AFAIK it doesn't actually *do* anything on Linux, and Linux doesn't require it.
I thought Thierry did this experiment and decided it wasn't necessary:
Could you detail what would be visible PCI bus topology if I remove the emulated PCI host bridge? (And keeping one PCI-to-PCI bridge per PCIe interface) ?
I'm just trying to understand what it would look like, in terms of "lspci -t" output, because for now, it's not clear to me how everything would fit together with the emulated host bridge.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 13 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines
Quoted 4 lines
I'm refering to your earlier question about what PCI IDs to use for the SW emulated devices. If there is no need for the host bridge then you only need 1 PCI ID (for the root port bridge) and you can probably fairly safely re-use the one in the Marvell config space of the HW.
Ah, ok, I see. But isn't a host bridge needed to bind all the PCI-to-PCI bridges under a single bus, in order to get the global resource assignment I was referring to?
Quoted 8 lines
Quoted 4 lines
The PCI-E spec requires it, but AFAIK it doesn't actually *do* anything on Linux, and Linux doesn't require it.
I thought Thierry did this experiment and decided it wasn't necessary:
Could you detail what would be visible PCI bus topology if I remove the emulated PCI host bridge? (And keeping one PCI-to-PCI bridge per PCIe interface) ?
So if this is what you have now...
-[0000:00]-+-00.0 <- Host bridge
+-10.0 <- Bridge
+-11.0 <- Bridge
+-12.0 <- BridgeThen removing the config space at 00.0 (ie the host bridge) will give you:
-[0000:00]-+-10.0 <- Bridge
+-11.0 <- Bridge
+-12.0 <- BridgeThe 'host bridge' isn't a bridge in the PCI-PCI sense, it is just a normal device@bus 0, device 0, function 0.
Jason
PATCH v2 7/27 PCI: Add software-emulated host bridge
Dear Jason Gunthorpe,
Quoted 10 lines · Jason Gunthorpe
So if this is what you have now...
-[0000:00]-+-00.0 <- Host bridge
+-10.0 <- Bridge +-11.0 <- Bridge +-12.0 <- BridgeThen removing the config space at 00.0 (ie the host bridge) will give you:
Quoted 6 lines
-[0000:00]-+-10.0 <- Bridge
+-11.0 <- Bridge +-12.0 <- BridgeThe 'host bridge' isn't a bridge in the PCI-PCI sense, it is just a normal device at bus 0, device 0, function 0.
Ok, thanks I'll try this!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 7/27 PCI: Add software-emulated host bridge
In reply to Jason GunthorpeQuoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Stephen Warren
Quoted 17 lines · Arnd Bergmann
Quoted 9 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]Signed-off-by Thierry Reding <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
Quoted 5 lines
Quoted 5 lines
At least on Tegra, there is no HW that exposes PCI configuration registers for the host bridge itself. Only the root ports have exposed PCI configuration registers. There was some debate re: whether a host bridge device needed to exist or not. This patch makes such a device exist if it's required.
Host bridges are not actually PCI devices on any architecture. The upstream side of a host bridge is by definition not on a PCI bus. On some architectures, it *looks* like the host bridge is a PCI device because it responds to PCI config accesses and you can get to configuration registers that way. But it isn't really; you can't enumerate host bridges by using normal PCI device enumeration because you have to somehow discover the root bus and the method of doing config accesses to it. That is all outside the scope of PCI. Even on the architectures where host bridges appear in PCI config space, the only reason that works is because we assume a config access mechanism that works for domain 0. We can't discover bridges in other domains without help.
Quoted 5 lines
If Linux will discover properly (I strongly suspect it does) without the host bridge, then I would say to ditch this...
The PCI-E standard requires a host bridge device, but if Linux doesn't require it then there is no reason to emulate one.
I agree that you don't need to emulate anything in the sense of making config space accessors as this patch does.
However, I think you *should* use pci_scan_root_bus() (maybe you do already; I haven't read all these patches), which requires that you know the configuration of the host bridge, i.e., the config access mechanism, the bus number range below the host bridge, and the I/O and MMIO apertures through the bridge. The PCI core builds a logical host bridge structure internally from that information, and that's all Linux really needs.
Bjorn
PATCH v2 7/27 PCI: Add software-emulated host bridge
Quoted 28 lines · Bjorn Helgaas
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Stephen Warren
Quoted 17 lines · Arnd Bergmann
Quoted 9 lines · Thomas Petazzoni
From: Thierry Reding <[email protected]>
[Thomas Petazzoni:
- Simplify capabilities handling.
- Move to a separate file.
- Fix mask used when writing a 4 bytes value.]Signed-off-by Thierry Reding <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
Quoted 5 lines
Quoted 5 lines
At least on Tegra, there is no HW that exposes PCI configuration registers for the host bridge itself. Only the root ports have exposed PCI configuration registers. There was some debate re: whether a host bridge device needed to exist or not. This patch makes such a device exist if it's required.
Quoted 4 lines
Host bridges are not actually PCI devices on any architecture. The upstream side of a host bridge is by definition not on a PCI bus. On some architectures, it *looks* like the host bridge is a PCI device because it responds to PCI config accesses and you can get to
Sure, you can't discover domains through any standard means, but once you have found a domain (notably a way to issue config transactions) then the PCI-E standard actually does place requirements on what config transactions should return:
- 0:00.0 is a host bridge config space.
- 0:XX.X will be one of:
- A root complex internal function, with some restrictions this is basically a PCI end device
- A PCI-PCI bridge with various mandatory capability headers. One of these must show up for every physical PCI-E link on the root complex.
This collection of stuff on bus 0 is called the 'root complex'. This is new in PCI-E, PCI-X and PCI didn't have such requirements.
SOC vendors are taking various liberties with their PCI-E implementations.
- nvidia followed the standard but did not include the host bridge at 0:00.0
- Marvell ignored everything about the root complex config space behavior :)
There are two patch sets in this subject, one for nvidia tegra and one for Marvell, both presenting to Linux a view of the HW that matches what the PCI-E spec describes - specifically that there is one domain, and each PCI-E link/controller shows up as a PCI-PCI bridge on bus 0.
In this model, there is no 'host bridge aperture' hardware, each PCI-E link has a dedicated aperture and control of that aperture is through the PCI-PCI bridge window registers, again as PCI-E specifies.
Jason
PATCH v2 7/27 PCI: Add software-emulated host bridge
In reply to Arnd BergmannDear Arnd Bergmann,
Quoted 6 lines · Arnd Bergmann
Not even a description why this is needed?
This patch (together with patch 8) seems like the most controversial one of the series, so you should better provide a really good reason why we would emulate something in software rather than using whatever hardware is there.
Hum, you're right. In fact, the very reason why I'm adding an emulated host bridge and emulated PCI-to-PCI bridges is simply because this was one of the main suggestion raised during the review of the first revision of this patch set.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
In reply to Thomas Petazzoni--- drivers/pci/Kconfig | 3 + drivers/pci/Makefile | 1 + drivers/pci/sw-pci-pci-bridge.c | 185 +++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 43 +++++++++ 4 files changed, 232 insertions(+) create mode 100644 drivers/pci/sw-pci-pci-bridge.c
drivers/pci/Kconfig +3 -0
@@ -122,3 +122,6 @@ config PCI_LABEL config PCI_SW_HOST_BRIDGE bool + +config PCI_SW_PCI_PCI_BRIDGE + bool
drivers/pci/Makefile +1 -0
@@ -17,6 +17,7 @@ obj-$(CONFIG_PCI_IOAPIC) += ioapic.o # Emulated PCI elements obj-$(CONFIG_PCI_SW_HOST_BRIDGE) += sw-host-bridge.o +obj-$(CONFIG_PCI_SW_PCI_PCI_BRIDGE) += sw-pci-pci-bridge.o # Build the PCI Hotplug drivers if we were asked to obj-$(CONFIG_HOTPLUG_PCI) += hotplug/
drivers/pci/sw-pci-pci-bridge.c new file +185 -0
185 lines not on this page · raw message
include/linux/pci.h +43 -0
47 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
drivers/pci/sw-pci-pci-bridge.c line 40
Quoted 4 lines · Thomas Petazzoni
drivers/pci/sw-pci-pci-bridge.c new file +4
-0+int pci_sw_pci_bridge_read (struct pci_sw_pci_bridge *bridge, +unsigned int where,int size,u32 *value) +{ +switch (where & ~3 ) {
It is not essential, but desirable, to report an Express Root Port capability for PCI-E bridges:
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <1us, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BC, TimeoutDis+ ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB
In the Marvell case, this capability can be constructed by pulling data from the the Express End Point capability of the PCI-E port:
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
This lets user space see the width/speed/etc state of the PCI-E link itself...
Jason
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Dear Jason Gunthorpe,
Thanks a lot for your quick feedback!
Quoted 2 lines · Jason Gunthorpe
It is not essential, but desirable, to report an Express Root Port capability for PCI-E bridges:
[...]
Quoted 2 lines
In the Marvell case, this capability can be constructed by pulling data from the the Express End Point capability of the PCI-E port:
I am not sure what you mean by "pulling". Do you mean that I should get informations from the real PCIe interface, from within the emulated PCI-to-PCI bridge implementation? This would unfortunately not be really nice, because until now, the PCI-to-PCI bridge emulation is clearly separated from the Marvell PCIe driver itself. Of course, it could register a hook or something like that, so that the emulated PCI-to-PCI bridge could potentially call back into the Marvell PCIe driver.
I'll have to dig a little bit more about this capability to see how it works exactly.
Thanks again for the feedback,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Quoted 11 lines · Thomas Petazzoni
Quoted 2 lines
In the Marvell case, this capability can be constructed by pulling data from the the Express End Point capability of the PCI-E port:
I am not sure what you mean by "pulling". Do you mean that I should get informations from the real PCIe interface, from within the emulated PCI-to-PCI bridge implementation? This would unfortunately not be really nice, because until now, the PCI-to-PCI bridge emulation is clearly separated from the Marvell PCIe driver itself. Of course, it could register a hook or something like that, so that the emulated PCI-to-PCI bridge could potentially call back into the Marvell PCIe driver.
Yes, a callback would be needed to the main driver and IIRC the driver can read/write the end port link info config regsiters via MMIO. They probably need a bit of massaging to be in root port format, but otherwise it should be straightforward..
Quoted 2 lines
I'll have to dig a little bit more about this capability to see how it works exactly.
All ports have registers to report and control the link, but the root port and end port versions are a bit different, so the goal is to read the end port formatted registers and map them into the root port format so that userspace can properly see the link state and configuration.
Jason
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Quoted 19 lines · Jason Gunthorpe
Quoted 11 lines · Thomas Petazzoni
Quoted 2 lines
In the Marvell case, this capability can be constructed by pulling data from the the Express End Point capability of the PCI-E port:
I am not sure what you mean by "pulling". Do you mean that I should get informations from the real PCIe interface, from within the emulated PCI-to-PCI bridge implementation? This would unfortunately not be really nice, because until now, the PCI-to-PCI bridge emulation is clearly separated from the Marvell PCIe driver itself. Of course, it could register a hook or something like that, so that the emulated PCI-to-PCI bridge could potentially call back into the Marvell PCIe driver.
Yes, a callback would be needed to the main driver and IIRC the driver can read/write the end port link info config regsiters via MMIO. They probably need a bit of massaging to be in root port format, but otherwise it should be straightforward..
Quoted 8 lines
Quoted 2 lines
I'll have to dig a little bit more about this capability to see how it works exactly.
All ports have registers to report and control the link, but the root port and end port versions are a bit different, so the goal is to read the end port formatted registers and map them into the root port format so that userspace can properly see the link state and configuration.
Isn't the thing being emulated here a host bridge, which "contains" the PCIe root ports underneath, which in turn "contain" the PCIe devices underneath? At least on Tegra, there is no host bridge device that exposes PCIe config registers, but the PCIe root ports do exist and do expose PCIe config registers...
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Quoted 29 lines · Stephen Warren
Quoted 19 lines · Jason Gunthorpe
Quoted 11 lines · Thomas Petazzoni
Quoted 2 lines
In the Marvell case, this capability can be constructed by pulling data from the the Express End Point capability of the PCI-E port:
I am not sure what you mean by "pulling". Do you mean that I should get informations from the real PCIe interface, from within the emulated PCI-to-PCI bridge implementation? This would unfortunately not be really nice, because until now, the PCI-to-PCI bridge emulation is clearly separated from the Marvell PCIe driver itself. Of course, it could register a hook or something like that, so that the emulated PCI-to-PCI bridge could potentially call back into the Marvell PCIe driver.
Yes, a callback would be needed to the main driver and IIRC the driver can read/write the end port link info config regsiters via MMIO. They probably need a bit of massaging to be in root port format, but otherwise it should be straightforward..
Quoted 8 lines
Quoted 2 lines
I'll have to dig a little bit more about this capability to see how it works exactly.
All ports have registers to report and control the link, but the root port and end port versions are a bit different, so the goal is to read the end port formatted registers and map them into the root port format so that userspace can properly see the link state and configuration.
Quoted 5 lines
Isn't the thing being emulated here a host bridge, which "contains" the PCIe root ports underneath, which in turn "contain" the PCIe devices underneath? At least on Tegra, there is no host bridge device that exposes PCIe config registers, but the PCIe root ports do exist and do expose PCIe config registers...
Patch #7 create a SW emulated host bridge, which tegra and marvell lack in HW.
Patch #8 creates a SW emulated root port bridge, which tegra has properly in HW, while Marvell doesn't.
Basically, on the Marvell chips, the PCI config space of the PCI complex is useless when used as a root complex - the config space is only usable when the device is configured as an end port.
Jason
Bjorn Helgaas
maintainerPATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
In reply to Thomas PetazzoniQuoted 8 lines · Thomas Petazzoni
Signed-off-by Thomas Petazzoni <[email protected]>--- drivers/pci/Kconfig | 3 + drivers/pci/Makefile | 1 + drivers/pci/sw-pci-pci-bridge.c | 185 +++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 43 +++++++++ 4 files changed, 232 insertions(+) create mode 100644 drivers/pci/sw-pci-pci-bridge.c
If you need this, it can be done in architecture code, can't it? It's true that there's nothing architecture-specific in this patch (other than the fact that ARM is the only arch that needs it), but I'm not sure there's anything useful for sharing here.
In fact, it seems like what you're after is not so much an *emulated* bridge that has no corresponding hardware, as it is a wrapper that presents a standard PCIe interface to hardware that exists but doesn't conform to the PCIe spec. If you really do need to ultimately connect this pci_sw_pci_bridge to a piece of hardware, that will certainly be arch-specific.
Quoted 269 lines
drivers/pci/Kconfig +3
-0@@ -122,3 +122,6 @@ config PCI_LABEL config PCI_SW_HOST_BRIDGE bool + +config PCI_SW_PCI_PCI_BRIDGE + booldrivers/pci/Makefile +1
-0@@ -17,6 +17,7 @@ obj-$(CONFIG_PCI_IOAPIC) += ioapic.o # Emulated PCI elements obj-$(CONFIG_PCI_SW_HOST_BRIDGE) += sw-host-bridge.o +obj-$(CONFIG_PCI_SW_PCI_PCI_BRIDGE) += sw-pci-pci-bridge.o # Build the PCI Hotplug drivers if we were asked to obj-$(CONFIG_HOTPLUG_PCI) += hotplug/drivers/pci/sw-pci-pci-bridge.c new file +185
-0@@ -0,0 +1,185 @@ +/* +* Implementation of a simple emulated PCI-to-PCI bridge. +* +* Thierry Reding <[email protected]> +* Thomas Petazzoni <[email protected]> +* +* This file is licensed under the terms of the GNU General Public +* License version 2. This program is licensed "as is" without any +* warranty of any kind, whether express or implied. +*/ + +#include <linux/kernel.h> +#include <linux/init.h> +#include <linux/pci.h> +#include <linux/module.h> + +int pci_sw_pci_bridge_init (struct pci_sw_pci_bridge *bridge) +{ +if (!bridge) +return -EINVAL ; + +memset (bridge,0 ,sizeof (struct pci_sw_pci_bridge )); + + bridge->status =PCI_STATUS_CAP_LIST ; + bridge->class =PCI_CLASS_BRIDGE_PCI ; + bridge->header_type =PCI_HEADER_TYPE_BRIDGE ; + bridge->cache_line_size =0x10 ; + +/* We support 32 bits I/O addressing */ + bridge->iobase =PCI_IO_RANGE_TYPE_32 ; + bridge->iolimit =PCI_IO_RANGE_TYPE_32 ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (pci_sw_pci_bridge_init); + +int pci_sw_pci_bridge_read (struct pci_sw_pci_bridge *bridge, +unsigned int where,int size,u32 *value) +{ +switch (where & ~3 ) { +case PCI_VENDOR_ID : + *value = bridge->device <<16 | bridge->vendor; +break ; + +case PCI_COMMAND : + *value = bridge->status <<16 | bridge->command; +break ; + +case PCI_CLASS_REVISION : + *value = bridge->class <<16 | bridge->interface <<8 | + bridge->revision; +break ; + +case PCI_CACHE_LINE_SIZE : + *value = bridge->bist <<24 | bridge->header_type <<16 | + bridge->latency_timer <<8 | bridge->cache_line_size; +break ; + + case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1: + *value = bridge->bar[((where & ~3 ) -PCI_BASE_ADDRESS_0 ) /4 ]; +break ; + +case PCI_PRIMARY_BUS : + *value = (bridge->secondary_latency_timer <<24 | + bridge->subordinate_bus <<16 | + bridge->secondary_bus <<8 | + bridge->primary_bus); +break ; + +case PCI_IO_BASE : + *value = (bridge->secondary_status <<16 | + bridge->iolimit <<8 | + bridge->iobase); +break ; + +case PCI_MEMORY_BASE : + *value = (bridge->memlimit <<16 | bridge->membase); +break ; + +case PCI_PREF_MEMORY_BASE : + *value = (bridge->prefmemlimit <<16 | bridge->prefmembase); +break ; + +case PCI_PREF_BASE_UPPER32 : + *value = bridge->prefbaseupper; +break ; + +case PCI_PREF_LIMIT_UPPER32 : + *value = bridge->preflimitupper; +break ; + +case PCI_IO_BASE_UPPER16 : + *value = (bridge->iolimitupper <<16 | bridge->iobaseupper); +break ; + +case PCI_ROM_ADDRESS1 : + *value =0 ; +break ; + +default : + *value =0xffffffff ; +return PCIBIOS_BAD_REGISTER_NUMBER ; + } + +if (size ==2 ) + *value = (*value >> (8 * (where &3 ))) &0xffff ; +else if (size ==1 ) + *value = (*value >> (8 * (where &3 ))) &0xff ; + +return PCIBIOS_SUCCESSFUL ; +} +EXPORT_SYMBOL_GPL (pci_sw_pci_bridge_read); + +int pci_sw_pci_bridge_write (struct pci_sw_pci_bridge *bridge, +unsigned int where,int size,u32 value) +{ +u32 mask, reg; +int err; + +if (size ==4 ) + mask =0x0 ; +else if (size ==2 ) + mask = ~(0xffff << ((where &3 ) *8 )); +else if (size ==1 ) + mask = ~(0xff << ((where &3 ) *8 )); +else +return PCIBIOS_BAD_REGISTER_NUMBER ; + + err =pci_sw_pci_bridge_read (bridge, where & ~3 ,4 , ®); +if (err) +return err; + + value = (reg & mask) | value << ((where &3 ) *8 ); + +switch (where & ~3 ) { +case PCI_COMMAND : + bridge->command = value &0xffff ; + bridge->status = value >>16 ; +break ; + + case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1: + bridge->bar[((where & ~3 ) -PCI_BASE_ADDRESS_0 ) /4 ] = value; +break ; + +case PCI_IO_BASE : +/* +* We also keep bit 1 set, it is a read-only bit that +* indicates we support 32 bits addressing for the +* I/O +*/ + bridge->iobase = (value &0xff ) |PCI_IO_RANGE_TYPE_32 ; + bridge->iolimit = ((value >>8 ) &0xff ) |PCI_IO_RANGE_TYPE_32 ; + bridge->secondary_status = value >>16 ; +break ; + +case PCI_MEMORY_BASE : + bridge->membase = value &0xffff ; + bridge->memlimit = value >>16 ; +break ; + +case PCI_PREF_MEMORY_BASE : + bridge->prefmembase = value &0xffff ; + bridge->prefmemlimit = value >>16 ; +break ; + +case PCI_PREF_BASE_UPPER32 : + bridge->prefbaseupper = value; +break ; + +case PCI_PREF_LIMIT_UPPER32 : + bridge->preflimitupper = value; +break ; + +case PCI_IO_BASE_UPPER16 : + bridge->iobaseupper = value &0xffff ; + bridge->iolimitupper = value >>16 ; +break ; + +default : +break ; + } + +return PCIBIOS_SUCCESSFUL ; +} +EXPORT_SYMBOL_GPL (pci_sw_pci_bridge_write);include/linux/pci.h +43
-1@@ -1864,4 +1864,47 @@ extern int pci_sw_host_bridge_read(struct pci_sw_host_bridge *bridge,extern int pci_sw_host_bridge_write (struct pci_sw_host_bridge *bridge,unsigned int where,int size,u32 value); +struct pci_sw_pci_bridge { +u16 vendor; +u16 device; +u16 command; +u16 status; +u16 class; +u8 interface; +u8 revision; +u8 bist; +u8 header_type; +u8 latency_timer; +u8 cache_line_size; +u32 bar[2 ]; +u8 primary_bus; +u8 secondary_bus; +u8 subordinate_bus; +u8 secondary_latency_timer; +u8 iobase; +u8 iolimit; +u16 secondary_status; +u16 membase; +u16 memlimit; +u16 prefmembase; +u16 prefmemlimit; +u32 prefbaseupper; +u32 preflimitupper; +u16 iobaseupper; +u16 iolimitupper; +u8 cappointer; +u8 reserved1; +u16 reserved2; +u32 romaddr; +u8 intline; +u8 intpin; +u16 bridgectrl; +}; + +extern int pci_sw_pci_bridge_init (struct pci_sw_pci_bridge *bridge); +extern int pci_sw_pci_bridge_read (struct pci_sw_pci_bridge *bridge, +unsigned int where,int size,u32 *value); +extern int pci_sw_pci_bridge_write (struct pci_sw_pci_bridge *bridge, +unsigned int where,int size,u32 value); +#endif /* LINUX_PCI_H */ --1.7.9.5
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Quoted 4 lines · Bjorn Helgaas
If you need this, it can be done in architecture code, can't it? It's true that there's nothing architecture-specific in this patch (other than the fact that ARM is the only arch that needs it), but I'm not sure there's anything useful for sharing here.
Since we're moving the host bridge code to drivers/pci/host now, I think this code should live in the same place. It's entirely possible that it will be shared between arch/arm and arch/arm64, although I would hope that we can do away with the emulated bridge code entirely.
More generally speaking, this is infrastructure code, and for any piece of infrastructure my rule is
* don't add platform specific infrastructure if it can be done at the architecture level
* don't add architecture specific infrastructure if it can be written in an architecture independent way
* don't add global infrastructure if you can use or extend the existing infrastructure.
Quoted 6 lines
In fact, it seems like what you're after is not so much an emulated bridge that has no corresponding hardware, as it is a wrapper that presents a standard PCIe interface to hardware that exists but doesn't conform to the PCIe spec. If you really do need to ultimately connect this
pci_sw_pci_bridgeto a piece of hardware, that will certainly be arch-specific.
As Jason Gunthorpe suggested, we might not need this at all, if the Linux PCI code can be convinced not to need a configuration space for the devices that in case of the Marvell hardware don't provide one.
Arnd
PATCH v2 8/27 pci: implement an emulated PCI-to-PCI bridge
Quoted 10 lines · Arnd Bergmann
Quoted 4 lines · Bjorn Helgaas
If you need this, it can be done in architecture code, can't it? It's true that there's nothing architecture-specific in this patch (other than the fact that ARM is the only arch that needs it), but I'm not sure there's anything useful for sharing here.
Since we're moving the host bridge code to
drivers/pci/hostnow, I think this code should live in the same place. It's entirely possible that it will be shared betweenarch/armandarch/arm64, although I would hope that we can do away with the emulated bridge code entirely.
This sounds right to me, this is part of the host bridge driver for various Marvell SOCs, so these days it should live in the drivers/pci/host or related, not arch/arm.
Quoted 11 lines
Quoted 6 lines
In fact, it seems like what you're after is not so much an emulated bridge that has no corresponding hardware, as it is a wrapper that presents a standard PCIe interface to hardware that exists but doesn't conform to the PCIe spec. If you really do need to ultimately connect this
pci_sw_pci_bridgeto a piece of hardware, that will certainly be arch-specific.As Jason Gunthorpe suggested, we might not need this at all, if the Linux PCI code can be convinced not to need a configuration space for the devices that in case of the Marvell hardware don't provide one.
To be clear, that isn't what I was talking about.. Just to clarify a few things in the last couple emails:
The PCI 'host bridge configuration space' software emulation code in patch #7 is not necessary. Bjorn and Thierry both confirm this.
In several places when Bjorn/Arnd talked about a 'host bridge' this is referring to (more or less) the PCI host *driver* and its attachment to the kernel interfaces. Specifically a configuration access mechanism and the resource ranges to allocate against. It has nothing to do with the bus 0, device 0, function 0 host bridge config space.
Arnd's suggestion to use multiple domains would be broadly equivilent to the first iteration of this driver - essentially the driver would manage one link and there would be multiple instances. This gets us back to where Thomas started - there is currently no code to do cross domain resource allocation, and static allocation is not possible with so many links on the chip.
Bjorn is quite right, the purpose of the PCI-PCI SW layer is to bind the non-standard registers in the Marvell SOC to the standard PCI-E config interface so the kernel can control it normally. This corrects what is, IMHO, a defect in the Marvell hardware.
The alternative is to add some kind of cross-domain resource allocation (or similar) to the PCI core code - however this would *only* be required to support hardware broken in the same way as Marvell, so I feel a bit leery about doing that kind of work before we know if other chips require this. (early on in the discussion there was some thought that Tegra might also be similary broken, but it turned out to be pretty much fine, with a bit of driver work)
So, I still think using a SW layer to provide a compliant PCI-PCI bridge configuration space for the Marvell hardware is the best way forward..
Jason
PATCH v2 9/27 pci: infrastructure to add drivers in drivers/pci/host
In reply to Thomas PetazzoniAs agreed by the community, PCI host drivers will now be stored in drivers/pci/host. This commit adds this directory and the related Kconfig/Makefile changes to allow new drivers to be added in this directory.
--- drivers/pci/Kconfig | 2 ++ drivers/pci/Makefile | 3 +++ drivers/pci/host/Kconfig | 4 ++++ 3 files changed, 9 insertions(+) create mode 100644 drivers/pci/host/Kconfig
drivers/pci/Kconfig +2 -0
@@ -125,3 +125,5 @@ config PCI_SW_HOST_BRIDGE config PCI_SW_PCI_PCI_BRIDGE bool + +source "drivers/pci/host/Kconfig"
drivers/pci/Makefile +3 -0
@@ -71,3 +71,6 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_OF) += of.o ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG + +# PCI host controller drivers +obj-y += host/
drivers/pci/host/Kconfig new file +4 -0
@@ -0,0 +1,4 @@ +menu "PCI host controller drivers" + depends on PCI + +endmenu
Signature · 1 line
1.7.9.5
PATCH v2 10/27 arm: mvebu: fix address-cells in mpic DT node
In reply to Thomas PetazzoniThere is no need to have a #address-cells property in the MPIC Device Tree node, and more than that, having it confuses the of_irq_map_raw() logic, which will be used by the Marvell PCIe driver.
--- arch/arm/boot/dts/armada-370-xp.dtsi | 1 - 1 file changed, 1 deletion(-)
arch/arm/boot/dts/armada-370-xp.dtsi +0 -1
@@ -31,7 +31,6 @@ mpic: interrupt-controller at d0020000 { compatible = "marvell,mpic"; #interrupt-cells = <1>; - #address-cells = <1>; #size-cells = <1>; interrupt-controller; };
Signature · 1 line
1.7.9.5
PATCH v2 11/27 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
In reply to Thomas PetazzoniThe Armada 370 has two gatable clocks for each PCIe interface, and we want both of them to be enabled. We therefore make one of the two clocks a child of the other, as we did for the sataX and sataXlnk clocks on Armada XP.
--- drivers/clk/mvebu/clk-gating-ctrl.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/clk/mvebu/clk-gating-ctrl.c +2 -2
@@ -119,8 +119,8 @@ static const struct mvebu_soc_descr __initconst armada_370_gating_descr[] = { { "pex1_en", NULL, 2 }, { "ge1", NULL, 3 }, { "ge0", NULL, 4 }, - { "pex0", NULL, 5 }, - { "pex1", NULL, 9 }, + { "pex0", "pex0_en", 5 }, + { "pex1", "pex1_en", 9 }, { "sata0", NULL, 15 }, { "sdio", NULL, 17 }, { "tdm", NULL, 25 },
Signature · 1 line
1.7.9.5
PATCH v2 11/27 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
Quoted 8 lines · Thomas Petazzoni
The Armada 370 has two gatable clocks for each PCIe interface, and we want both of them to be enabled. We therefore make one of the two clocks a child of the other, as we did for the sataX and sataXlnk clocks on Armada XP.
drivers/clk/mvebu/clk-gating-ctrl.c static const struct mvebu_soc_descr __initconst armada_370_gating_descr[] = { · line 123
Quoted 7 lines
drivers/clk/mvebu/clk-gating-ctrl.c +2
-2@@ -119,8 +119,8 @@ static const struct mvebu_soc_descr __initconst armada_370_gating_descr[] = { { "pex1_en", NULL, 2 }, { "ge1", NULL, 3 }, { "ge0", NULL, 4 }, - { "pex0", NULL, 5 }, - { "pex1", NULL, 9 }, + { "pex0", "pex0_en", 5 }, + { "pex1", "pex1_en", 9 },
I must admit, I know nothing about struct mvebu_soc_descr, but I'm having a hard time seeing how that code change makes one of those clock a parent of the other, since the pex0 entry doesn't reference anything "pex1"-related, nor vice-versa. Is more explanation in the commit message warranted here?
PATCH v2 11/27 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
Dear Stephen Warren,
Quoted 5 lines · Stephen Warren
I must admit, I know nothing about struct
mvebu_soc_descr, but I'm having a hard time seeing how that code change makes one of those clock a parent of the other, since the pex0 entry doesn't reference anything "pex1"-related, nor vice-versa. Is more explanation in the commit message warranted here?
See the definition of mvebu_soc_descr:
struct mvebu_soc_descr {
const char *name;
const char *parent;
int bit_idx;
};It simply registers the pex0 clock with the pex0_en clock as its parents. Those clocks are normal gatable clocks, registered with clk_register_gate(). This ensures that whenever the pex0 clock is enabled, its parent clock pex0_en gets enabled as well. We do the same for SATA clocks on Armada XP, for example:
static const struct mvebu_soc_descr __initconst armada_xp_gating_descr[] = { { "audio", NULL, 0 }, [...]
{ "sata0lnk", NULL, 14 },
{ "sata0", "sata0lnk", 15 },
[...]
{ "sata1lnk", NULL, 29 },
{ "sata1", "sata1lnk", 30 },
{ }
};Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 11/27 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
Quoted 18 lines · Thomas Petazzoni
Dear Stephen Warren,
Quoted 5 lines · Stephen Warren
I must admit, I know nothing about struct
mvebu_soc_descr, but I'm having a hard time seeing how that code change makes one of those clock a parent of the other, since the pex0 entry doesn't reference anything "pex1"-related, nor vice-versa. Is more explanation in the commit message warranted here?See the definition of
mvebu_soc_descr:struct mvebu_soc_descr { const char *name; const char *parent; int bit_idx; };
Quoted 4 lines
It simply registers the pex0 clock with the
pex0_enclock as its parents. Those clocks are normal gatable clocks, registered withclk_register_gate(). This ensures that whenever the pex0 clock is enabled, its parent clockpex0_engets enabled as well.
Oh I see; I was confused by the patch description. The two clocks being made child/parent are the two clocks for a port, and this relationship is set up for each port; for some reason I thought there was a requirement to make one port's clock a child of the other port's clock.
PATCH v2 11/27 clk: mvebu: create parent-child relation for PCIe clocks on Armada 370
Dear Stephen Warren,
Quoted 4 lines · Stephen Warren
Oh I see; I was confused by the patch description. The two clocks being made child/parent are the two clocks for a port, and this relationship is set up for each port; for some reason I thought there was a requirement to make one port's clock a child of the other port's clock.
Aah, ok, I understand the confusion now. Re-reading my commit log, I'm not sure where the confusion comes from, but english is not my native language, so maybe something that sounds clear to me is not clear in reality. I'll rephrase the commit log to make sure this confusion is clarified.
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 12/27 clk: mvebu: add more PCIe clocks for Armada XP
In reply to Thomas PetazzoniThe current revision of the datasheet only mentions the gatable clocks for the PCIe 0.0, 0.1, 0.2 and 0.3 interfaces, and forgot to mention the ones for the PCIe 1.0, 1.1, 1.2, 1.3, 2.0 and 3.0 interfaces. After confirmation with Marvell engineers, this patch adds the missing gatable clocks for those PCIe interfaces.
It also changes the name of the previously existing PCIe gatable clocks, in order to match the naming using the datasheets.
--- drivers/clk/mvebu/clk-gating-ctrl.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/clk/mvebu/clk-gating-ctrl.c +10 -4
@@ -137,10 +137,14 @@ static const struct mvebu_soc_descr __initconst armada_xp_gating_descr[] = { { "ge2", NULL, 2 }, { "ge1", NULL, 3 }, { "ge0", NULL, 4 }, - { "pex0", NULL, 5 }, - { "pex1", NULL, 6 }, - { "pex2", NULL, 7 }, - { "pex3", NULL, 8 }, + { "pex00", NULL, 5 }, + { "pex01", NULL, 6 }, + { "pex02", NULL, 7 }, + { "pex03", NULL, 8 }, + { "pex10", NULL, 9 }, + { "pex11", NULL, 10 }, + { "pex12", NULL, 11 }, + { "pex13", NULL, 12 }, { "bp", NULL, 13 }, { "sata0lnk", NULL, 14 }, { "sata0", "sata0lnk", 15 },
@@ -152,6 +156,8 @@ static const struct mvebu_soc_descr __initconst armada_xp_gating_descr[] = { { "xor0", NULL, 22 }, { "crypto", NULL, 23 }, { "tdm", NULL, 25 }, + { "pex20", NULL, 26 }, + { "pex30", NULL, 27 }, { "xor1", NULL, 28 }, { "sata1lnk", NULL, 29 }, { "sata1", "sata1lnk", 30 },
Signature · 1 line
1.7.9.5
PATCH v2 13/27 arm: plat-orion: introduce WIN_CTRL_ENABLE in address mapping code
In reply to Thomas PetazzoniInstead of hardcoding "1" as being the bit value to enable an address decoding window, introduce and use a WIN_CTRL_ENABLE definition.
--- arch/arm/plat-orion/addr-map.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm/plat-orion/addr-map.c +3 -1
@@ -38,6 +38,7 @@ EXPORT_SYMBOL_GPL(mv_mbus_dram_info);* CPU Address Decode Windows registers */ #define WIN_CTRL_OFF 0x0000 +#define WIN_CTRL_ENABLE BIT(0)#define WIN_BASE_OFF 0x0004#define WIN_REMAP_LO_OFF 0x0008#define WIN_REMAP_HI_OFF 0x000c
@@ -79,7 +80,8 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, } base_high = base &0xffff0000 ; - ctrl = ((size -1 ) &0xffff0000 ) | (attr <<8 ) | (target <<4 ) |1 ; + ctrl = ((size -1 ) &0xffff0000 ) | (attr <<8 ) | (target <<4 ) | +WIN_CTRL_ENABLE ;writel (base_high, addr +WIN_BASE_OFF );writel (ctrl, addr +WIN_CTRL_OFF );
Signature · 1 line
1.7.9.5
PATCH v2 14/27 arm: plat-orion: refactor the orion_disable_wins() function
In reply to Thomas PetazzoniIn the address decoding code, the orion_disable_wins() function is used at boot time to disable all address decoding windows, before configuring only the ones that are needed. This allows to make sure that no configuration is left from the bootloader.
As a preparation for the introduction of address decoding window allocation/deallocation function, we refactor this function into an orion_disable_cpu_win() which disables a single window.
The orion_config_wins() function is changed to call orion_disable_cpu_win() in a loop, to preserve an identical behavior.
--- arch/arm/plat-orion/addr-map.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
arch/arm/plat-orion/addr-map.c +17 -18
@@ -95,6 +95,19 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, } } +static void __init orion_disable_cpu_win (const struct orion_addr_map_cfg *cfg, +const int win) +{ +void __iomem *addr = cfg->win_cfg_base (cfg, win); + +writel (0 , addr +WIN_BASE_OFF ); +writel (0 , addr +WIN_CTRL_OFF ); +if (cfg->cpu_win_can_remap (cfg, win)) { +writel (0 , addr +WIN_REMAP_LO_OFF ); +writel (0 , addr +WIN_REMAP_HI_OFF ); + } +} +/* * Configure a number of windows. */
@@ -108,36 +121,22 @@ static void __init orion_setup_cpu_wins(const struct orion_addr_map_cfg * cfg, } } -static void __init orion_disable_wins (const struct orion_addr_map_cfg * cfg) -{ -void __iomem *addr; -int i; - -for (i =0 ; i < cfg->num_wins; i++) { - addr = cfg->win_cfg_base (cfg, i); - -writel (0 , addr +WIN_BASE_OFF ); -writel (0 , addr +WIN_CTRL_OFF ); -if (cfg->cpu_win_can_remap (cfg, i)) { -writel (0 , addr +WIN_REMAP_LO_OFF ); -writel (0 , addr +WIN_REMAP_HI_OFF ); - } - } -} -/* * Disable, clear and configure windows. */ void __init orion_config_wins (struct orion_addr_map_cfg * cfg,const struct orion_addr_map_info *info) { +int win; +if (!cfg->cpu_win_can_remap) cfg->cpu_win_can_remap = orion_cpu_win_can_remap;if (!cfg->win_cfg_base) cfg->win_cfg_base = orion_win_cfg_base; -orion_disable_wins (cfg); +for (win =0 ; win < cfg->num_wins; win++) +orion_disable_cpu_win (cfg, win);if (info)orion_setup_cpu_wins (cfg, info);
Signature · 1 line
1.7.9.5
PATCH v2 15/27 arm: plat-orion: introduce orion_{alloc, free}_cpu_win() functions
In reply to Thomas PetazzoniIn the address decoding code, we implement two new functions:orion_alloc_cpu_win() and orion_free_cpu_win(). The first function finds an unused address decoding window, and configures it according to the given arguments (in terms of base address, size, target, attributes). The second function frees an address decoding window, given a physical base address.
Those two new functions will be used by the PCIe code, which needs to dynamically register address decoding windows depending on the PCIe devices that are detected.
The orion_free_cpu_win() function is only here to handle error cases in the PCIe devices initialization, in the normal case, address decoding windows are never freed.
--- arch/arm/plat-orion/addr-map.c | 50 +++++++++++++++++++++++++++ arch/arm/plat-orion/include/plat/addr-map.h | 7 ++++ 2 files changed, 57 insertions(+)
arch/arm/plat-orion/addr-map.c +50 -0
@@ -109,6 +109,56 @@ static void __init orion_disable_cpu_win(const struct orion_addr_map_cfg *cfg, }/* +* Find an unused address decoding window, and enable it according to +* the arguments passed (base, size, target, attributes, remap). +*/ +int __init orion_alloc_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base,const u32 size, +const u8 target,const u8 attr,const int remap) +{ +int win; + +for (win =0 ; win < cfg->num_wins; win++) { +void __iomem *addr = cfg->win_cfg_base (cfg, win); +u32 ctrl =readl (addr +WIN_CTRL_OFF ); +if (!(ctrl &WIN_CTRL_ENABLE )) +break ; + } + +/* No more windows available */ +if (win == cfg->num_wins) +return -ENOMEM ; + +orion_setup_cpu_win (cfg, win, base, size, target, attr, remap); +return 0 ; +} + +/* +* Free an address decoding window, given its base address. +*/ +int __init orion_free_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base) +{ +int win; + +for (win =0 ; win < cfg->num_wins; win++) { +void __iomem *addr = cfg->win_cfg_base (cfg, win); +u32 winbase =readl (addr +WIN_BASE_OFF ); +u32 ctrl =readl (addr +WIN_CTRL_OFF ); + +if (!(ctrl &WIN_CTRL_ENABLE )) +continue ; + +if (winbase == (base &0xffff0000 )) { +orion_disable_cpu_win (cfg, win); +return 0 ; + } + } + +return -EINVAL ; +} + +/* * Configure a number of windows. */ static void __init orion_setup_cpu_wins(const struct orion_addr_map_cfg * cfg,
arch/arm/plat-orion/include/plat/addr-map.h +7 -0
@@ -49,6 +49,13 @@ void __init orion_setup_cpu_win(const struct orion_addr_map_cfg *cfg, const u32 size, const u8 target, const u8 attr, const int remap); +int __init orion_alloc_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base,const u32 size, +const u8 target,const u8 attr,const int remap); + +int __init orion_free_cpu_win (const struct orion_addr_map_cfg *cfg, +const u32 base); +void __init orion_setup_cpu_mbus_target (const struct orion_addr_map_cfg *cfg,const void __iomem *ddr_window_cpu_base);#endif
Signature · 1 line
1.7.9.5
PATCH v2 16/27 arm: mvebu: add functions to alloc/free PCIe decoding windows
In reply to Thomas PetazzoniThis commit adds two functions armada_370_xp_alloc_pcie_window() and armada_370_xp_free_pcie_window() that respectively allocate and free an address decoding window pointing to either a memory or I/O region of a PCIe device.
Those functions will be used by the PCIe driver to create and remove those regions depending on the PCIe devices that are detected.
--- arch/arm/mach-mvebu/addr-map.c | 87 +++++++++++++++++++++++++-- arch/arm/mach-mvebu/common.h | 1 + arch/arm/mach-mvebu/include/mach/addr-map.h | 8 +++ 3 files changed, 92 insertions(+), 4 deletions(-) create mode 100644 arch/arm/mach-mvebu/include/mach/addr-map.h
arch/arm/mach-mvebu/addr-map.c +83 -4
@@ -24,14 +24,10 @@#define ARMADA_XP_TARGET_DEV_BUS 1#define ARMADA_XP_ATTR_DEV_BOOTROM 0x1D#define ARMADA_XP_TARGET_ETH1 3 -#define ARMADA_XP_TARGET_PCIE_0_2 4#define ARMADA_XP_TARGET_ETH0 7 -#define ARMADA_XP_TARGET_PCIE_1_3 8#define ARMADA_370_TARGET_DEV_BUS 1#define ARMADA_370_ATTR_DEV_BOOTROM 0x1D -#define ARMADA_370_TARGET_PCIE_0 4 -#define ARMADA_370_TARGET_PCIE_1 8#define ARMADA_WINDOW_8_PLUS_OFFSET 0x90#define ARMADA_SDRAM_ADDR_DECODING_OFFSET 0x180
@@ -89,6 +85,89 @@ static struct __initdata orion_addr_map_cfg addr_map_cfg = { .win_cfg_base = armada_cfg_base, }; +#ifdef CONFIG_PCI +/* +* This structure and the following arrays allow to map a PCIe (port, +* lane) tuple to the corresponding (target, attribute) tuple needed +* to configure an address decoding window for the given PCIe (port, +* lane). +*/ +struct pcie_mapping { +int port; +int lane; +u8 target; +u8 attr; +}; + +struct pcie_mapping armada_xp_pcie_mappings[] = { + { .port =0 , .lane =0 , .target =4 , .attr =0xE0 }, + { .port =0 , .lane =1 , .target =4 , .attr =0xD0 }, + { .port =0 , .lane =2 , .target =4 , .attr =0xB0 }, + { .port =0 , .lane =3 , .target =4 , .attr =0x70 }, + { .port =1 , .lane =0 , .target =8 , .attr =0xE0 }, + { .port =1 , .lane =1 , .target =8 , .attr =0xD0 }, + { .port =1 , .lane =2 , .target =8 , .attr =0xB0 }, + { .port =1 , .lane =3 , .target =8 , .attr =0x70 }, + { .port =2 , .lane =0 , .target =4 , .attr =0xF0 }, + { .port =3 , .lane =0 , .target =8 , .attr =0xF0 }, + { .port =-1 }, +}; + +struct pcie_mapping armada_370_pcie_mappings[] = { + { .port =0 , .lane =0 , .target =4 , .attr =0xE0 }, + { .port =1 , .lane =0 , .target =8 , .attr =0xE0 }, + { .port =-1 }, +}; + +/* +* This function sets up a new address decoding window at the given +* base address, pointing to the given PCIe interface (through +* pcie_port and pcie_lane). +*/ +int __init armada_370_xp_alloc_pcie_window (int pcie_port,int pcie_lane, +unsigned long base,u32 size, +int type) +{ +struct pcie_mapping *mapping, *mappings; +u8 target, attr; + +if (of_machine_is_compatible ("marvell,armadaxp" )) + mappings = armada_xp_pcie_mappings; +else if (of_machine_is_compatible ("marvell,armada370" )) + mappings = armada_370_pcie_mappings; +else +return -ENODEV ; + +for (mapping = mappings; mapping->port !=-1 ; mapping++) +if (mapping->port == pcie_port && mapping->lane == pcie_lane) +break ; + +if (mapping->port ==-1 ) +return -ENODEV ; + + target = mapping->target; + attr = mapping->attr; + +/* +* Bit 3 of the attributes indicates that it is a +* memory region, as opposed to an I/O region +*/ +if (type ==IORESOURCE_MEM ) + attr |= (1 <<3 ); + +return orion_alloc_cpu_win (&addr_map_cfg, base, size, target, attr,-1 ); +} + +/* +* Frees an address decoding window previously set up by +* armada_370_xp_setup_pcie_window(). +*/ +int __init armada_370_xp_free_pcie_window (unsigned long base) +{ +return orion_free_cpu_win (&addr_map_cfg, base); +} +#endif +static int __init armada_setup_cpu_mbus (void ) {struct device_node *np;
arch/arm/mach-mvebu/common.h +1 -0
@@ -25,4 +25,5 @@ int armada_370_xp_coherency_init(void);int armada_370_xp_pmsu_init (void );void armada_xp_secondary_startup (void );extern struct smp_operations armada_xp_smp_ops; +#endif
arch/arm/mach-mvebu/include/mach/addr-map.h new file +8 -0
@@ -0,0 +1,8 @@ +#ifndef MVEBU_ADDR_MAP_H +#define MVEBU_ADDR_MAP_H + +int armada_370_xp_alloc_pcie_window (int pcie_port,int pcie_lane, +unsigned long base,u32 size,int type); +int armada_370_xp_free_pcie_window (unsigned long base); + +#endif /* MVEBU_ADDR_MAP_H */
Signature · 1 line
1.7.9.5
PATCH v2 17/27 arm: plat-orion: make common PCIe code usable on mvebu
In reply to Thomas Petazzonimvebu is a new-style Orion platform, so it only selects PLAT_ORION, but not PLAT_ORION_LEGACY. It will however need the common PCIe code from plat-orion, so make this code available for PLAT_ORION platforms as a whole, and not only PLAT_ORION_LEGACY platforms.
We also take this opportunity to build the PCIe code only when CONFIG_PCI is enabled.
--- arch/arm/plat-orion/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm/plat-orion/Makefile +2 -1
@@ -4,7 +4,8 @@ ccflags-$(CONFIG_ARCH_MULTIPLATFORM) := -I$(srctree)/$(src)/include obj-y += addr-map.o +obj-$(CONFIG_PCI) += pcie.o orion-gpio-$(CONFIG_GENERIC_GPIO) += gpio.o -obj-$(CONFIG_PLAT_ORION_LEGACY) += irq.o pcie.o time.o common.o mpp.o +obj-$(CONFIG_PLAT_ORION_LEGACY) += irq.o time.o common.o mpp.o obj-$(CONFIG_PLAT_ORION_LEGACY) += $(orion-gpio-y)
Signature · 1 line
1.7.9.5
PATCH v2 18/27 arm: plat-orion: add more flexible PCI configuration space read/write functions
In reply to Thomas PetazzoniThe existing orion_pcie_rd_conf() and orion_pcie_wr_conf() functions provided by plat-orion/pcie.c are nice to read and write the PCI configuration space of a device, but they unfortunately assume that the bus number and slot number at which a device is visible at the Linux software level is the same as the bus number and slot number at the hardware level.
However, with the usage of the emulated PCI host bridge and emulated PCI-to-PCI bridges, this is not the case: bus number 0 is the emulated bus on which the emulated PCI-to-PCI bridges sit, so from the Linux point of view, the real busses start at bus 1, but from a hardware point of view, they start at bus 0.
So, we cannot use the existing orion_pcie_rd_conf() and orion_pcie_wr_conf() implementations, which take their bus number directly from a given pci_bus structure. Instead, we add lower-level variants, orion_pcie_rd_conf_bus() and orion_pcie_wr_conf_bus() that take a bus number as argument. The existing orion_pcie_rd_conf() and orion_pcie_wr_conf() functions are implemented on top of the new *_bus() variants.
Those *_bus() variants will be used by the Marvell Armada 370/XP PCIe driver.
--- arch/arm/plat-orion/include/plat/pcie.h | 4 ++++ arch/arm/plat-orion/pcie.c | 26 ++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)
arch/arm/plat-orion/include/plat/pcie.h +4 -0
@@ -21,12 +21,16 @@ int orion_pcie_get_local_bus_nr(void __iomem *base);void orion_pcie_set_local_bus_nr (void __iomem *base,int nr);void orion_pcie_reset (void __iomem *base);void orion_pcie_setup (void __iomem *base); +int orion_pcie_rd_conf_bus (void __iomem *base,u32 busn, +u32 devfn,int where,int size,u32 *val);int orion_pcie_rd_conf (void __iomem *base,struct pci_bus *bus,u32 devfn,int where,int size,u32 *val);int orion_pcie_rd_conf_tlp (void __iomem *base,struct pci_bus *bus,u32 devfn,int where,int size,u32 *val);int orion_pcie_rd_conf_wa (void __iomem *wa_base,struct pci_bus *bus,u32 devfn,int where,int size,u32 *val); +int orion_pcie_wr_conf_bus (void __iomem *base,u32 busn, +u32 devfn,int where,int size,u32 val);int orion_pcie_wr_conf (void __iomem *base,struct pci_bus *bus,u32 devfn,int where,int size,u32 val);
arch/arm/plat-orion/pcie.c +20 -6
@@ -203,10 +203,10 @@ void __init orion_pcie_setup(void __iomem *base)writel (mask, base +PCIE_MASK_OFF ); } -int orion_pcie_rd_conf (void __iomem *base,struct pci_bus *bus, -u32 devfn,int where,int size,u32 *val) +int orion_pcie_rd_conf_bus (void __iomem *base,u32 busn,u32 devfn, +int where,int size,u32 *val) { - writel(PCIE_CONF_BUS(bus->number) | + writel(PCIE_CONF_BUS(busn) | PCIE_CONF_DEV(PCI_SLOT(devfn)) | PCIE_CONF_FUNC(PCI_FUNC(devfn)) | PCIE_CONF_REG(where) | PCIE_CONF_ADDR_EN,
@@ -222,6 +222,13 @@ int orion_pcie_rd_conf(void __iomem *base, struct pci_bus *bus,return PCIBIOS_SUCCESSFUL ; } +int orion_pcie_rd_conf (void __iomem *base,struct pci_bus *bus, +u32 devfn,int where,int size,u32 *val) +{ +return orion_pcie_rd_conf_bus (base, bus->number, devfn, + where, size, val); +} +int orion_pcie_rd_conf_tlp (void __iomem *base,struct pci_bus *bus,u32 devfn,int where,int size,u32 *val) {
@@ -261,12 +268,12 @@ int orion_pcie_rd_conf_wa(void __iomem *wa_base, struct pci_bus *bus,return PCIBIOS_SUCCESSFUL ; } -int orion_pcie_wr_conf (void __iomem *base,struct pci_bus *bus, -u32 devfn,int where,int size,u32 val) +int orion_pcie_wr_conf_bus (void __iomem *base,u32 busn, +u32 devfn,int where,int size,u32 val) {int ret =PCIBIOS_SUCCESSFUL ; - writel(PCIE_CONF_BUS(bus->number) | + writel(PCIE_CONF_BUS(busn) | PCIE_CONF_DEV(PCI_SLOT(devfn)) | PCIE_CONF_FUNC(PCI_FUNC(devfn)) | PCIE_CONF_REG(where) | PCIE_CONF_ADDR_EN,
@@ -284,3 +291,10 @@ int orion_pcie_wr_conf(void __iomem *base, struct pci_bus *bus,return ret; } + +int orion_pcie_wr_conf (void __iomem *base,struct pci_bus *bus, +u32 devfn,int where,int size,u32 val) +{ +return orion_pcie_wr_conf_bus (base, bus->number, devfn, + where, size, val); +}
Signature · 1 line
1.7.9.5
PATCH v2 18/27 arm: plat-orion: add more flexible PCI configuration space read/write functions
Quoted 5 lines · Thomas Petazzoni
However, with the usage of the emulated PCI host bridge and emulated PCI-to-PCI bridges, this is not the case: bus number 0 is the emulated bus on which the emulated PCI-to-PCI bridges sit, so from the Linux point of view, the real busses start at bus 1, but from a hardware point of view, they start at bus 0.
Hum.. This is a bit funny sounding, can you confirm..
The bus number programmed into all the end points must match the Linux number. Ie the PCI-E Link Description register of end point devices must report the same bus number as Linux. PCI-E devices learn their bus number by capturing the bus number from type 0 configuration transactions.
For the most part config transactions issued to the PCI-E controllers should be type 0 transactions with a bus number that matches what Linux is setting.
The only time I think you'd ever see bus number 0 is when accessing the config space of the Marvell PCI-E controller end port. But, I also think you can avoid doing these transactions by just accessing the MMIO versions of those registers..
Jason
PATCH v2 18/27 arm: plat-orion: add more flexible PCI configuration space read/write functions
Dear Jason Gunthorpe,
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Thomas Petazzoni
However, with the usage of the emulated PCI host bridge and emulated PCI-to-PCI bridges, this is not the case: bus number 0 is the emulated bus on which the emulated PCI-to-PCI bridges sit, so from the Linux point of view, the real busses start at bus 1, but from a hardware point of view, they start at bus 0.
Hum.. This is a bit funny sounding, can you confirm..
Might be yes, but IIRC, when I try to enumerate the devices in the PCIe interface 0 (from a hardware point of view), passing a bus number of 1 in the PCI configuration space access registers, then it simply doesn't work.
Quoted 2 lines
The bus number programmed into all the end points must match the Linux number. Ie the PCI-E Link Description register of end point devices
What is this PCI-E Link Description register ? Where is it located ?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 18/27 arm: plat-orion: add more flexible PCI configuration space read/write functions
Quoted 11 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
Quoted 8 lines · Jason Gunthorpe
Quoted 5 lines · Thomas Petazzoni
However, with the usage of the emulated PCI host bridge and emulated PCI-to-PCI bridges, this is not the case: bus number 0 is the emulated bus on which the emulated PCI-to-PCI bridges sit, so from the Linux point of view, the real busses start at bus 1, but from a hardware point of view, they start at bus 0.
Quoted 6 lines
Quoted 1 line
Hum.. This is a bit funny sounding, can you confirm..
Might be yes, but IIRC, when I try to enumerate the devices in the PCIe interface 0 (from a hardware point of view), passing a bus number of 1 in the PCI configuration space access registers, then it simply doesn't work.
Hurm. The trick is you need the chip to issue a type 0 request. The Marvell docs say this happens automatically basd on the 'internal bus number'
The only other reference to bus number is in the PCI Express Status Register (41A04), so that probably needs to be set to the subordinate bus number of the bridge.
Quoted 4 lines
Quoted 2 lines
The bus number programmed into all the end points must match the Linux number. Ie the PCI-E Link Description register of end point devices
What is this PCI-E Link Description register ? Where is it located ?
Hum, looks like this is only for root complex links not end devices - PCI-X had a register for this but it seems to have been removed in PCI-E.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniThis driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the hw_pci operations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and the pci_ops operations that are used by the PCI core to read/write the configuration space of PCI devices.
Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
In addition, this driver enumerates the different PCIe slots, and for those having a device plugged in, it sets up the necessary address decoding windows, using the new armada_370_xp_alloc_pcie_window() function from mach-mvebu/addr-map.c.
--- .../devicetree/bindings/pci/armada-370-xp-pcie.txt | 175 +++++++ drivers/pci/host/Kconfig | 6 + drivers/pci/host/Makefile | 4 + drivers/pci/host/pci-mvebu.c | 500 ++++++++++++++++++++ 4 files changed, 685 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt create mode 100644 drivers/pci/host/Makefile create mode 100644 drivers/pci/host/pci-mvebu.c
Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt new file +175 -0
@@ -0,0 +1,175 @@ +* Marvell Armada 370/XP PCIe interfaces + +Mandatory properties: +- compatible: must be "marvell,armada-370-xp-pcie" +- status: either "disabled" or "okay" +- #address-cells, set to <3> +- #size-cells, set to <2> +- #interrupt-cells, set to <1> +- bus-range: PCI bus numbers covered +- ranges: standard PCI-style address ranges, describing the PCIe + registers for each PCIe interface, and then ranges for the PCI + memory and I/O regions. +- interrupt-map-mask and interrupt-map are standard PCI Device Tree + properties to describe the interrupts associated to each PCI + interface. + +In addition, the Device Tree node must have sub-nodes describing each +PCIe interface, having the following mandatory properties: +- reg: the address and size of the PCIe registers (translated + addresses according to the ranges property of the parent) +- clocks: the clock associated to this PCIe interface +- marvell,pcie-port: the physical PCIe port number +- status: either "disabled" or "okay" + +and the following optional properties: +- marvell,pcie-lane: the physical PCIe lane number, for ports having + multiple lanes. If this property is not found, we assume that the + value is 0. + +Example: + +pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */ + 0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */ + 0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */ + 0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>; + interrupt-map = <0x0800 0 0 1 &mpic 58 + 0x1000 0 0 1 &mpic 59 + 0x1800 0 0 1 &mpic 60 + 0x2000 0 0 1 &mpic 61 + 0x2800 0 0 1 &mpic 62 + 0x3000 0 0 1 &mpic 63 + 0x3800 0 0 1 &mpic 64 + 0x4000 0 0 1 &mpic 65 + 0x4800 0 0 1 &mpic 99 + 0x5000 0 0 1 &mpic 103>; + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 0,1 { + device_type = "pciex"; + reg = <0x1000 0 0xd0044000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 6>; + status = "disabled"; + }; + + pcie at 0,2 { + device_type = "pciex"; + reg = <0x1800 0 0xd0048000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 7>; + status = "disabled"; + }; + + pcie at 0,3 { + device_type = "pciex"; + reg = <0x2000 0 0xd004C000 0 0xC000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 8>; + status = "disabled"; + }; + + pcie at 1,0 { + device_type = "pciex"; + reg = <0x2800 0 0xd0080000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 9>; + status = "disabled"; + }; + + pcie at 1,1 { + device_type = "pciex"; + reg = <0x3000 0 0xd0084000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 10>; + status = "disabled"; + }; + + pcie at 1,2 { + device_type = "pciex"; + reg = <0x3800 0 0xd0088000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 11>; + status = "disabled"; + }; + + pcie at 1,3 { + device_type = "pciex"; + reg = <0x4000 0 0xd008C000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 12>; + status = "disabled"; + }; + pcie at 2,0 { + device_type = "pciex"; + reg = <0x4800 0 0xd0042000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <2>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 26>; + status = "disabled"; + }; + + pcie at 3,0 { + device_type = "pciex"; + reg = <0x5000 0 0xd0082000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <3>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 27>; + status = "disabled"; + }; +};
drivers/pci/host/Kconfig +6 -0
@@ -1,4 +1,10 @@ menu "PCI host controller drivers" depends on PCI +config PCI_MVEBU + bool "Marvell EBU PCIe controller" + depends on ARCH_MVEBU + select PCI_SW_HOST_BRIDGE + select PCI_SW_PCI_PCI_BRIDGE + endmenu
drivers/pci/host/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +ccflags-$(CONFIG_PCI_MVEBU) += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/include
drivers/pci/host/pci-mvebu.c new file +500 -0
500 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 21 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the
hw_pcioperations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and thepci_opsoperations that are used by the PCI core to read/write the configuration space of PCI devices.Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
In addition, this driver enumerates the different PCIe slots, and for those having a device plugged in, it sets up the necessary address decoding windows, using the new
armada_370_xp_alloc_pcie_window()function frommach-mvebu/addr-map.c.
Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt line 5
Quoted 4 lines
Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt +3
-0+Mandatory properties: +- compatible: must be "marvell,armada-370-xp-pcie" +- status: either "disabled" or "okay"
status is a standard DT property; I certainly wouldn't expect its presence to be mandatory (there's a defined default), nor would I expect each device's binding to redefine this property.
Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt line 22
Quoted 4 lines
Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt +3
-0+In addition, the Device Tree node must have sub-nodes describing each +PCIe interface, having the following mandatory properties: +- marvell,pcie-port: the physical PCIe port number
Should the standardized cell-index property be used here instead? Or, perhaps that property is deprecated/discouraged...
Quoted 1 line
+- status: either "disabled" or "okay"
Similar comment as above.
drivers/pci/host/Makefile line 4
Quoted 6 lines
drivers/pci/host/Makefile +4
-0+obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +ccflags-$(CONFIG_PCI_MVEBU) += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/include
That seems a little dangerous w.r.t. multi-platform zImage. Can the required headers be moved out to somewhere more public to avoid this?
drivers/pci/host/pci-mvebu.c line 28
Quoted 9 lines
drivers/pci/host/pci-mvebu.c +7
-0+/* +* Those are the product IDs used for the emulated PCI Host bridge and +* emulated PCI-to-PCI bridges. They are temporary until we get +* official IDs assigned. +*/ +#define MARVELL_EMULATED_HOST_BRIDGE_ID 4141 +#define MARVELL_EMULATED_PCI_PCI_BRIDGE_ID 4242
I assume that means we can't merge this driver yet. The cover letter mentioned a desire to merge this for 3.9; there's not much time to get official IDs assigned, then.
drivers/pci/host/pci-mvebu.c line 496
Quoted 7 lines
drivers/pci/host/pci-mvebu.c +7
-0+static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall (mvebu_pcie_init);
Why isn't that just platform_driver_register()?
Quoted 1 line
+MODULE_LICENSE("GPL");
"GPL v2".
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Stephen Warren,
Quoted 7 lines · Stephen Warren
Quoted 3 lines
+Mandatory properties:
+- compatible: must be "marvell,armada-370-xp-pcie" +- status: either "disabled" or "okay"status is a standard DT property; I certainly wouldn't expect its presence to be mandatory (there's a defined default), nor would I expect each device's binding to redefine this property.
Ok.
Quoted 4 lines
Quoted 1 line
+- marvell,pcie-port: the physical PCIe port number
Should the standardized cell-index property be used here instead? Or, perhaps that property is deprecated/discouraged...
The problem is that I need two identifiers, the pcie-port and pcie-lane, and it would be strange to have one referenced as cell-index, and the other one as marvell,pcie-lane, no? Unless of course we can put two numbers in the cell-index property, but a quick grep in Documentation/devicetree/bindings/ seems to indicate that all users of cell-index use it with a single identifier.
Just tell me what to do here, I don't have a strong opinion on this.
Quoted 9 lines
drivers/pci/host/Makefile line 4
Quoted 6 lines
drivers/pci/host/Makefile +4
-0+obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +ccflags-$(CONFIG_PCI_MVEBU) += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/includeThat seems a little dangerous w.r.t. multi-platform zImage. Can the required headers be moved out to somewhere more public to avoid this?
Why is this dangerous for multi-platform zImage? For this specific driver only, some SoC-specific headers are used. I don't think it prevents another PCI driver (such as the Tegra one) from being built into the same kernel image, no?
Also, this is kind of a temporary solution, because I can't fix all the problems in just the PCI patch series without making it horribly large. The reason why we need those include paths at the moment are:
* For plat-orion/include, because of pcie.h that provides functions common to all Marvell SoCs regarding PCI. Ultimately, all the Marvell SoCs that use this common code should be migrated over to the DT-capable PCI driver that is submitted through this patch series. This will take a bit of time, and is too complex to do in one shot, together with the introduction of the driver itself. So ultimately, all the code in plat-orion/pcie.c will migrate into drivers/pci/host/pci-mvebu.c, and this plat-orion/include/plat/pcie.h file should disappear, removing the need for this special header path.
* For mach-mvebu/include, because of the addr-map.h header that provides functions related to address decoding windows. This is also likely to evolve quite significantly when we'll make the PCI driver being used by the other Marvell SoC families (Kirkwood, Dove, Orion5x), and when this work is done, we can think of having a public header in include/linux that exposes the address decoding APIs, once it has stabilized a bit across the different Marvell SoC families.
So, the bottom line is: yes I know those include paths are not nice, but I don't think they prevent multiplatform builds, and they are a temporary solution until we convert more Marvell SoC families to this new PCI driver.
Quoted 11 lines
drivers/pci/host/Makefile
Quoted 7 lines
drivers/pci/host/Makefile +7
-0+/* + * Those are the product IDs used for the emulated PCI Host bridge and + * emulated PCI-to-PCI bridges. They are temporary until we get + * official IDs assigned. + */ +#define MARVELL_EMULATED_HOST_BRIDGE_ID 4141 +#define MARVELL_EMULATED_PCI_PCI_BRIDGE_ID 4242I assume that means we can't merge this driver yet. The cover letter mentioned a desire to merge this for 3.9; there's not much time to get official IDs assigned, then.
I am working on getting real IDs assigned. For now, I'd like to work on getting all other issues fixed, and have this only problem remaining.
Quoted 10 lines
drivers/pci/host/Makefile
Quoted 8 lines
drivers/pci/host/Makefile +7
-0+static int mvebu_pcie_init(void) +{ + return platform_driver_probe(&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall(mvebu_pcie_init);Why isn't that just
platform_driver_register()?
I didn't test recently, but with my first version of the patch set, having an initialization as late as module_init() was too late. Some PCI fixup code was being executed *before* we get the opportunity of initializing the PCI driver, and it was crashing the kernel. I can provide more details if you want.
Quoted 3 lines
Quoted 1 line
+MODULE_LICENSE("GPL");
"GPL v2".
Sure.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Thomas Petazzoni
On Mon, 28 Jan 2013 15:21:45 -0700, Stephen Warren wrote:
[...]
Quoted 10 lines
Quoted 9 lines
drivers/pci/host/pci-mvebu.c line 496
Quoted 7 lines
drivers/pci/host/pci-mvebu.c new file +7
-0+static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall (mvebu_pcie_init);Why isn't that just
platform_driver_register()?
Quoted 5 lines
I didn't test recently, but with my first version of the patch set, having an initialization as late as
module_init()was too late. Some PCI fixup code was being executed *before* we get the opportunity of initializing the PCI driver, and it was crashing the kernel. I can provide more details if you want.
Does this patch perhaps fix this crash?
http://patchwork.ozlabs.org/patch/210870/
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130129/579b43da/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Thierry Reding,
Quoted 3 lines · Thierry Reding
Does this patch perhaps fix this crash?
http://patchwork.ozlabs.org/patch/210870/
I'll test it, thanks for the notice!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingDear Thierry Reding,
Quoted 9 lines · Thierry Reding
Quoted 5 lines
I didn't test recently, but with my first version of the patch set, having an initialization as late as
module_init()was too late. Some PCI fixup code was being executed *before* we get the opportunity of initializing the PCI driver, and it was crashing the kernel. I can provide more details if you want.Does this patch perhaps fix this crash?
http://patchwork.ozlabs.org/patch/210870/
I investigated a bit more, and managed to reproduce my crash even with your patch applied. And indeed, my crash is really unrelated to the pcibios function disappearing. Here is the kernel panic (and a short analysis afterwards) :
Unhandled fault: external abort on non-linefetch (0x1008) at 0xe0910010
Internal error: : 1008 [#1] SMP ARM Modules linked in: CPU: 0 Not tainted (3.8.0-rc5-00029-g80e55fd-dirty #1303) PC is at quirk_usb_handoff_xhci+0x5c/0x284 LR is at ioremap_pte_range+0x84/0xdc pc : [<c022717c>] lr : [<c0150944>] psr: a0000013
sp : df82bce8 ip : df81c000 fp : c0e09dac
r10: 00008000 r9 : 00000000 r8 : de935000
r7 : e0910000 r6 : c03f0ce0 r5 : de935000 r4 : de935000 r3 : 01c801c8 r2 : 00000000 r1 : 42007e13 r0 : e0910000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c5387d Table: 1e95c019 DAC: 00000015 Process swapper/0 (pid: 1, stack limit = 0xdf82a238)
Stack: (0xdf82bce8 to 0xdf82c000)
bce0: de935060 c01702f4 de935000 de935000 de935000 c03f0ce0
bd00: c0e220dc c02277f8 de935060 c02277d4 c03f0ce0 c0227858 c03f0ce0 c0178d98
bd20: c01a855c c0312dd0 00000000 00000000 de936f6c c0312ed0 c0e0f7cc de935000
bd40: de936c14 de936c00 df82bde8 00000001 de916668 c016a958 00000000 de935000
bd60: de936c14 c016ab5c de935800 de910014 de936c00 c016abcc 00000000 00000001
bd80: de910000 c016d234 de916668 de9166d0 de916640 00000000 df82be14 c017a93c
bda0: de916668 df82be14 df82bde8 c0012ef8 f3cec23f c0251c90 c0019da0 df805340
bdc0: f3cec23f df82bde8 c0e426e8 df82be14 df802b00 de912a90 00000060 df89e400
bde0: 00000002 c00131dc df82bde8 df82bde8 c0e454f0 00000000 de9166d0 de912af0
be00: df802b00 c0442010 df89e410 00000000 00000000 c0e0fcac 00000001 df82be54
be20: c04420b4 c017a90c 00000000 00000000 00000000 c0442054 c1000000 c8ffffff
be40: c124b5f0 00000200 00000000 00000000 00000000 de9166d0 df89e410 c0e43e48
be60: c0e0fc6c df89e410 00000000 c0e0fc6c c04563d4 c017a7c4 00000000 c01a8c24
be80: c01a8c0c c01a78e8 df89e410 c0e0fc6c df89e444 00000000 c043023c c01a7bd8
bea0: c0e0fc6c 00000000 c01a7b4c c01a632c df8067d8 df85a474 c0e0fc6c c0e170c8
bec0: de916740 c01a7228 c03d28bc c0e0fc6c c0e0fc6c df82a000 c0e220c0 00000000
bee0: c043023c c01a80d8 00000000 c0e0fc58 df82a000 c0e220c0 00000000 c043023c
bf00: c017a7c4 c01a8e1c c044fc80 df82a000 c0e220c0 c00086d4 c040e208 00000006
bf20: 0000007b c017a7c4 0000007b 00000006 00000006 c043023c c124bb15 00000000
bf40: c0e08194 c044fc80 00000006 c044fc60 c0e220c0 c043023c c04563d4 0000007b
bf60: 00000000 c04308b0 00000006 00000006 c043023c 00000000 c0456128 c0456128
bf80: 00000000 00000000 00000000 00000000 00000000 c0430958 00000000 00000000
bfa0: c03156c8 c03156d0 00000000 c000dfd8 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 2254c9ef c6885425
[<c022717c>] (quirk_usb_handoff_xhci+0x5c/0x284) from [<c02277d4>] (quirk_usb_early_handoff.part.1+0xc0/0xe4) [<c02277d4>] (quirk_usb_early_handoff.part.1+0xc0/0xe4) from [<c0178d98>] (pci_do_fixups+0xc4/0x17c) [<c0178d98>] (pci_do_fixups+0xc4/0x17c) from [<c016a958>] (pci_bus_add_device+0x14/0x60) [<c016a958>] (pci_bus_add_device+0x14/0x60) from [<c016ab5c>] (pci_bus_add_devices+0x44/0x128) [<c016ab5c>] (pci_bus_add_devices+0x44/0x128) from [<c016abcc>] (pci_bus_add_devices+0xb4/0x128) [<c016abcc>] (pci_bus_add_devices+0xb4/0x128) from [<c016d234>] (pci_scan_root_bus+0x7c/0xcc) [<c016d234>] (pci_scan_root_bus+0x7c/0xcc) from [<c017a93c>] (mvebu_pcie_scan_bus+0x30/0x3c) [<c017a93c>] (mvebu_pcie_scan_bus+0x30/0x3c) from [<c0012ef8>] (pcibios_init_hw+0x5c/0x15c) [<c0012ef8>] (pcibios_init_hw+0x5c/0x15c) from [<c00131dc>] (pci_common_init+0x44/0xc4) [<c00131dc>] (pci_common_init+0x44/0xc4) from [<c0442010>] (mvebu_pcie_probe+0x360/0x3a4) [<c0442010>] (mvebu_pcie_probe+0x360/0x3a4) from [<c01a8c24>] (platform_drv_probe+0x18/0x1c) [<c01a8c24>] (platform_drv_probe+0x18/0x1c) from [<c01a78e8>] (really_probe+0x60/0x1e0) [<c01a78e8>] (really_probe+0x60/0x1e0) from [<c01a7bd8>] (__driver_attach+0x8c/0x90) [<c01a7bd8>] (__driver_attach+0x8c/0x90) from [<c01a632c>] (bus_for_each_dev+0x50/0x7c) [<c01a632c>] (bus_for_each_dev+0x50/0x7c) from [<c01a7228>] (bus_add_driver+0x168/0x22c) [<c01a7228>] (bus_add_driver+0x168/0x22c) from [<c01a80d8>] (driver_register+0x78/0x144) [<c01a80d8>] (driver_register+0x78/0x144) from [<c01a8e1c>] (platform_driver_probe+0x18/0xac) [<c01a8e1c>] (platform_driver_probe+0x18/0xac) from [<c00086d4>] (do_one_initcall+0x34/0x174) [<c00086d4>] (do_one_initcall+0x34/0x174) from [<c04308b0>] (do_basic_setup+0x90/0xc4) [<c04308b0>] (do_basic_setup+0x90/0xc4) from [<c0430958>] (kernel_init_freeable+0x74/0x10c) [<c0430958>] (kernel_init_freeable+0x74/0x10c) from [<c03156d0>] (kernel_init+0x8/0xe4) [<c03156d0>] (kernel_init+0x8/0xe4) from [<c000dfd8>] (ret_from_fork+0x14/0x3c)
Code: e3a02000 ebf7c092 e2507000 0afffff7 (e5973010) ---[ end trace 834a6081748c17ef ]--- Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
Basically, the problem comes from the fact that the USB XHCI code has an early PCI fixup, that ioremap() the PCI device memory BAR and makes access to it. Unfortunately, this fixup is called during pcibios_init_hw(), at a point where the Marvell PCIe driver haven't yet had a chance to set up the address decoding windows (the Linux PCI core hasn't even configured the emulated PCI-to-PCI bridges, so I don't know where the PCI devices will sit).
Due to this, the first access to the PCI device memory by this early fixup triggers an exception, and the kernel panics.
For some reason, moving the Marvell PCIe driver initialization at the subsys_initcall() level works around the problem, but I'm not sure why, since it is actually the driver initialization that ends up calling the PCI fixup code. But clearly, with the PCIe initialization done at subsys_initcall() time, the PCIe is initialized, and then a lot later the PCI fixup is executed.
Ideas welcome.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Thomas Petazzoni
Dear Thierry Reding,
Quoted 6 lines · Thierry Reding
Quoted 5 lines
I didn't test recently, but with my first version of the patch set, having an initialization as late as
module_init()was too late. Some PCI fixup code was being executed *before* we get the opportunity of initializing the PCI driver, and it was crashing the kernel. I can provide more details if you want.
Quoted 8 lines
Quoted 3 lines
Does this patch perhaps fix this crash?
http://patchwork.ozlabs.org/patch/210870/I investigated a bit more, and managed to reproduce my crash even with your patch applied. And indeed, my crash is really unrelated to the pcibios function disappearing. Here is the kernel panic (and a short analysis afterwards) :
Hi Thomas,
Can you post the entire dmesg log, ideally with CONFIG_PCI_DEBUG=y? That should have more information about the enumeration process, including what we think the XHCI BARs are and the apertures leading to them.
The PCI core assumes that we know the host bridge apertures up front, and I'm not sure that is true on your platform, so maybe we'll need some changes to accommodate that.
Bjorn
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Bjorn Helgaas,
Quoted 4 lines · Bjorn Helgaas
Can you post the entire dmesg log, ideally with
CONFIG_PCI_DEBUG=y? That should have more information about the enumeration process, including what we think the XHCI BARs are and the apertures leading to them.
Sure, see below.
Quoted 3 lines
The PCI core assumes that we know the host bridge apertures up front, and I'm not sure that is true on your platform, so maybe we'll need some changes to accommodate that.
In this hardware, we need to set up the address decoding windows. So there shouldn't be any access to a PCI device memory or I/O region until the addresses have been assigned in the PCI-to-PCI bridge.
Note that I am know setting up the address decoding window as soon as the address is written into the PCI-to-PCI bridge. I am no longer waiting the end of enumeration process, and then go through the PCI-to-PCI bridge registers to configure them.
The system tested below is an Armada 370, it has only two PCIe links. One is connected to a XHCI USB controller, the other one to an Intel Wireless NIC.
First the dmesg when module_init() is used, which shows the crash:
===================================================================
Booting Linux on physical CPU 0x0
Linux version 3.8.0-rc5-00029-g80e55fd-dirty (thomas at skate) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #1313 SMP Thu Feb 7 16:53:32 CET 2013
CPU: ARMv7 Processor [561f5811] revision 1 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Machine: Marvell Armada 370/XP (Device Tree), model: Globalscale Mirabox
bootconsole [earlycon0] enabled
Memory policy: ECC disabled, Data cache writeback
PERCPU: Embedded 7 pages/cpu @c128b000 s6464 r8192 d14016 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyS0,115200 earlyprintk
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)__ex_table already sorted, skipping sort
Memory: 512MB = 512MB total
Memory: 504860k/504860k available, 19428k reserved, 0K highmem
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc0467700 (4478 kB)
.init : 0xc0468000 - 0xc0e30940 (10019 kB)
.data : 0xc0e32000 - 0xc0e5f9c0 ( 183 kB)
.bss : 0xc0e5f9c0 - 0xc0e84fdc ( 150 kB)Hierarchical RCU implementation.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
NR_IRQS:16 nr_irqs:16 16
Aurora cache controller enabled
l2x0: 4 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a086302, Cache size: 262144 Bsched_clock: 32 bits at 18MHz, resolution 53ns, wraps every 229064ms
Console: colour dummy device 80x30
Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256)pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket -1, mpidr 0
Setting up static identity map for 0x351a28 - 0x351a80
Brought up 1 CPUs
SMP: Total of 1 processors activated (1196.85 BogoMIPS).
devtmpfs: initialized
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 1024 KiB pool for atomic coherent allocations
irq: Cannot allocate irq_descs @ IRQ27, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ63, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ96, assuming pre-allocated Initializing Coherency fabric
bio: create slab <bio-0> at 0
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
cfg80211: Calling CRDA to update world regulatory domain Switching to clocksource armada_370_xp_clocksource
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes) TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
msgmni has been set to 986
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
armada-370-pinctrl d0018000.pinctrl: registered pinctrl driver mvebu-pcie pcie-controller.1: PCIe0.0: link up
mvebu-pcie pcie-controller.1: PCIe1.0: link up
mvebu-pcie pcie-controller.1: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0xc0000000-0xc000ffff] pci_bus 0000:00: root bus resource [mem 0xc1000000-0xc8ffffff] pci_bus 0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
PCI: bus2: Fast back to back transfers disabled
Unhandled fault: external abort on non-linefetch (0x1008) at 0xe0910010
Internal error: : 1008 [#1] SMP ARM Modules linked in: CPU: 0 Not tainted (3.8.0-rc5-00029-g80e55fd-dirty #1313) PC is at quirk_usb_handoff_xhci+0x5c/0x284 LR is at ioremap_pte_range+0x84/0xdc pc : [<c023ba8c>] lr : [<c0150944>] psr: a0000013
sp : df82bce8 ip : df81c000 fp : c0e43dac
r10: 00008000 r9 : 00000000 r8 : de933000
r7 : e0910000 r6 : c04267a0 r5 : de933000 r4 : de933000 r3 : 01c801c8 r2 : 00000000 r1 : 42007e13 r0 : e0910000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c5387d Table: 1e960019 DAC: 00000015 Process swapper/0 (pid: 1, stack limit = 0xdf82a238)
Stack: (0xdf82bce8 to 0xdf82c000)
bce0: de933060 c01702f4 de933000 de933000 de933000 c04267a0
bd00: c0e5f9dc c023c108 de933060 c023c0e4 c04267a0 c023c168 c04267a0 c0178d98
bd20: c01a855c c033cab4 00000000 00000000 de935f6c c033cbb4 c0e497cc de933000
bd40: de935c14 de935c00 df82bde8 00000001 de919368 c016a958 00000000 de933000
bd60: de935c14 c016ab5c de933800 de918014 de935c00 c016abcc 00000000 00000001
bd80: de918000 c016d234 de919368 de9193d0 de919340 00000000 df82be14 c017a93c
bda0: de919368 df82be14 df82bde8 c0012ef8 f3cec23f c027b974 c0019da0 df805340
bdc0: f3cec23f df82bde8 c0e7ffe8 df82be14 df802b00 de917890 00000060 df89e400
bde0: 00000002 c00131dc df82bde8 df82bde8 c0e83318 00000000 de9193d0 de9178f0
be00: df802b00 c047a010 df89e410 00000000 00000000 c0e49cac 00000001 df82be54
be20: c047a0b4 c017a90c 00000000 00000000 00000000 c047a054 c1000000 c8ffffff
be40: c12885f0 00000200 00000000 00000000 00000000 de9193d0 df89e410 c0e81748
be60: c0e49c6c df89e410 00000000 c0e49c6c c048e9b8 c017a7c4 00000000 c01a8c24
be80: c01a8c0c c01a78e8 df89e410 c0e49c6c df89e444 00000000 c046823c c01a7bd8
bea0: c0e49c6c 00000000 c01a7b4c c01a632c df8067d8 df85a474 c0e49c6c c0e510c8
bec0: de919440 c01a7228 c0402548 c0e49c6c c0e49c6c df82a000 c0e5f9c0 00000000
bee0: c046823c c01a80d8 00000000 c0e49c58 df82a000 c0e5f9c0 00000000 c046823c
bf00: c017a7c4 c01a8e1c c0488260 df82a000 c0e5f9c0 c00086d4 c0444b18 00000006
bf20: 0000008b c017a7c4 0000008b 00000006 00000006 c046823c c1288b15 00000000
bf40: c0e42194 c0488260 00000006 c0488240 c0e5f9c0 c046823c c048e9b8 0000008b
bf60: 00000000 c04688b0 00000006 00000006 c046823c 00000000 c048e708 c048e708
bf80: 00000000 00000000 00000000 00000000 00000000 c0468958 00000000 00000000
bfa0: c033f3ac c033f3b4 00000000 c000dfd8 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 2254c9ef c6885425
[<c023ba8c>] (quirk_usb_handoff_xhci+0x5c/0x284) from [<c023c0e4>] (quirk_usb_early_handoff.part.1+0xc0/0xe4) [<c023c0e4>] (quirk_usb_early_handoff.part.1+0xc0/0xe4) from [<c0178d98>] (pci_do_fixups+0xc4/0x17c) [<c0178d98>] (pci_do_fixups+0xc4/0x17c) from [<c016a958>] (pci_bus_add_device+0x14/0x60) [<c016a958>] (pci_bus_add_device+0x14/0x60) from [<c016ab5c>] (pci_bus_add_devices+0x44/0x128) [<c016ab5c>] (pci_bus_add_devices+0x44/0x128) from [<c016abcc>] (pci_bus_add_devices+0xb4/0x128) [<c016abcc>] (pci_bus_add_devices+0xb4/0x128) from [<c016d234>] (pci_scan_root_bus+0x7c/0xcc) [<c016d234>] (pci_scan_root_bus+0x7c/0xcc) from [<c017a93c>] (mvebu_pcie_scan_bus+0x30/0x3c) [<c017a93c>] (mvebu_pcie_scan_bus+0x30/0x3c) from [<c0012ef8>] (pcibios_init_hw+0x5c/0x15c) [<c0012ef8>] (pcibios_init_hw+0x5c/0x15c) from [<c00131dc>] (pci_common_init+0x44/0xc4) [<c00131dc>] (pci_common_init+0x44/0xc4) from [<c047a010>] (mvebu_pcie_probe+0x360/0x3a4) [<c047a010>] (mvebu_pcie_probe+0x360/0x3a4) from [<c01a8c24>] (platform_drv_probe+0x18/0x1c) [<c01a8c24>] (platform_drv_probe+0x18/0x1c) from [<c01a78e8>] (really_probe+0x60/0x1e0) [<c01a78e8>] (really_probe+0x60/0x1e0) from [<c01a7bd8>] (__driver_attach+0x8c/0x90) [<c01a7bd8>] (__driver_attach+0x8c/0x90) from [<c01a632c>] (bus_for_each_dev+0x50/0x7c) [<c01a632c>] (bus_for_each_dev+0x50/0x7c) from [<c01a7228>] (bus_add_driver+0x168/0x22c) [<c01a7228>] (bus_add_driver+0x168/0x22c) from [<c01a80d8>] (driver_register+0x78/0x144) [<c01a80d8>] (driver_register+0x78/0x144) from [<c01a8e1c>] (platform_driver_probe+0x18/0xac) [<c01a8e1c>] (platform_driver_probe+0x18/0xac) from [<c00086d4>] (do_one_initcall+0x34/0x174) [<c00086d4>] (do_one_initcall+0x34/0x174) from [<c04688b0>] (do_basic_setup+0x90/0xc4) [<c04688b0>] (do_basic_setup+0x90/0xc4) from [<c0468958>] (kernel_init_freeable+0x74/0x10c) [<c0468958>] (kernel_init_freeable+0x74/0x10c) from [<c033f3b4>] (kernel_init+0x8/0xe4) [<c033f3b4>] (kernel_init+0x8/0xe4) from [<c000dfd8>] (ret_from_fork+0x14/0x3c)
Code: e3a02000 ebf76e4e e2507000 0afffff7 (e5973010) ---[ end trace 7097ba2281051df7 ]--- Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ===================================================================
And now, the dmesg when the PCIe driver is initialized at the subsys_initcall() level, no crash happens, and everything works fine.
===================================================================
Booting Linux on physical CPU 0x0
Linux version 3.8.0-rc5-00029-g80e55fd-dirty (thomas at skate) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #1314 SMP Thu Feb 7 16:57:39 CET 2013
CPU: ARMv7 Processor [561f5811] revision 1 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Machine: Marvell Armada 370/XP (Device Tree), model: Globalscale Mirabox
bootconsole [earlycon0] enabled
Memory policy: ECC disabled, Data cache writeback
PERCPU: Embedded 7 pages/cpu @c128b000 s6464 r8192 d14016 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyS0,115200 earlyprintk
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)__ex_table already sorted, skipping sort
Memory: 512MB = 512MB total
Memory: 504860k/504860k available, 19428k reserved, 0K highmem
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc0467700 (4478 kB)
.init : 0xc0468000 - 0xc0e30940 (10019 kB)
.data : 0xc0e32000 - 0xc0e5f9c0 ( 183 kB)
.bss : 0xc0e5f9c0 - 0xc0e84fdc ( 150 kB)Hierarchical RCU implementation.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
NR_IRQS:16 nr_irqs:16 16
Aurora cache controller enabled
l2x0: 4 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a086302, Cache size: 262144 Bsched_clock: 32 bits at 18MHz, resolution 53ns, wraps every 229064ms
Console: colour dummy device 80x30
Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256)pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket -1, mpidr 0
Setting up static identity map for 0x351a28 - 0x351a80
Brought up 1 CPUs
SMP: Total of 1 processors activated (1196.85 BogoMIPS).
devtmpfs: initialized
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 1024 KiB pool for atomic coherent allocations
irq: Cannot allocate irq_descs @ IRQ27, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ63, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ96, assuming pre-allocated Initializing Coherency fabric
bio: create slab <bio-0> at 0
mvebu-pcie pcie-controller.1: PCIe0.0: link up
mvebu-pcie pcie-controller.1: PCIe1.0: link up
mvebu-pcie pcie-controller.1: PCI host bridge to bus 0000:00pci_bus 0000:00: root bus resource [io 0xc0000000-0xc000ffff]pci_bus 0000:00: root bus resource [mem 0xc1000000-0xc8ffffff]pci_bus 0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
PCI: bus2: Fast back to back transfers disabled
Getting IRQ slot 1, pin 1...
Cannot get irq... 135
Getting IRQ slot 2, pin 1...
Cannot get irq... 135
Getting IRQ slot 1, pin 1...
Getting IRQ slot 2, pin 1...
pci 0000:00:01.0: BAR 8: assigned [mem 0xc1000000-0xc10fffff]
pci 0000:00:02.0: BAR 8: assigned [mem 0xc1100000-0xc11fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xc1000000-0xc1001fff 64bit]
pci 0000:00:01.0: PCI bridge to [bus 01]
pci 0000:00:01.0: bridge window [mem 0xc1000000-0xc10fffff]
pci 0000:02:00.0: BAR 0: assigned [mem 0xc1100000-0xc1107fff 64bit]
pci 0000:00:02.0: PCI bridge to [bus 02]
pci 0000:00:02.0: bridge window [mem 0xc1100000-0xc11fffff]
PCI: enabling device 0000:00:01.0 (0140 -> 0143)
PCI: enabling device 0000:00:02.0 (0140 -> 0143)
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
cfg80211: Calling CRDA to update world regulatory domain
Switching to clocksource armada_370_xp_clocksource
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
msgmni has been set to 986
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
armada-370-pinctrl d0018000.pinctrl: registered pinctrl drivermv_xor d0060800.xor: Marvell XOR drivermv_xor d0060800.xor: Marvell XOR: ( xor cpy )mv_xor d0060800.xor: Marvell XOR: ( xor fill cpy )mv_xor d0060900.xor: Marvell XOR drivermv_xor d0060900.xor: Marvell XOR: ( xor cpy )mv_xor d0060900.xor: Marvell XOR: ( xor fill cpy )
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled d0012000.serial: ttyS0 at MMIO 0xd0012000 (irq = 17) is a 8250
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
e1000e: Intel(R) PRO/1000 Network Driver - 2.1.4-k
e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
libphy: orion_mdio_bus: probed
mvneta d0070000.ethernet eth0: mac: fe:4b:b2:e2:64:ca
mvneta d0074000.ethernet eth1: mac: 56:dd:1a:9e:e0:aa
Intel(R) Wireless WiFi driver for Linux, in-tree:
Copyright(c) 2003-2012 Intel Corporation
iwlwifi 0000:01:00.0: pci_enable_msi failed(0Xffffffff)
iwlwifi 0000:01:00.0: loaded firmware version 8.83.5.1 build 33692 iwlwifi 0000:01:00.0: CONFIG_IWLWIFI_DEBUG disabled
iwlwifi 0000:01:00.0: CONFIG_IWLWIFI_DEBUGFS disabled
iwlwifi 0000:01:00.0: CONFIG_IWLWIFI_DEVICE_TRACING disabled
iwlwifi 0000:01:00.0: CONFIG_IWLWIFI_DEVICE_TESTMODE disabled
iwlwifi 0000:01:00.0: CONFIG_IWLWIFI_P2P enabled
iwlwifi 0000:01:00.0: Detected Intel(R) Ultimate N WiFi Link 5300 AGN, REV=0x24 iwlwifi 0000:01:00.0: L1 Disabled; Enabling L0Sxhci_hcd 0000:02:00.0: xHCI Host Controllerxhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 1 xhci_hcd 0000:02:00.0: irq 99, io mem 0xc1100000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detectedxhci_hcd 0000:02:00.0: xHCI Host Controllerxhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
mousedev: PS/2 mouse device common for all mice
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
TCP: cubic registered
VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6 /home/thomas/projets/linux-2.6/drivers/rtc/hctosys.c: unable to open rtc device (rtc0) Freeing init memory: 10016K
Starting logging: OK
Starting mdev...
usb 1-1: new high-speed USB device number 2 using xhci_hcd
scsi0 : usb-storage 1-1:1.0
Initializing random number generator... done.
Starting network...
Welcome to Buildroot
buildroot login:
===================================================================
Do not hesitate to ask if you need more details about this.
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 11 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 4 lines · Bjorn Helgaas
Can you post the entire dmesg log, ideally with
CONFIG_PCI_DEBUG=y? That should have more information about the enumeration process, including what we think the XHCI BARs are and the apertures leading to them.Sure, see below.
Quoted 7 lines
Quoted 3 lines
The PCI core assumes that we know the host bridge apertures up front, and I'm not sure that is true on your platform, so maybe we'll need some changes to accommodate that.
In this hardware, we need to set up the address decoding windows. So there shouldn't be any access to a PCI device memory or I/O region until the addresses have been assigned in the PCI-to-PCI bridge.
I think this is the path where the crash happens (this is the same as the backtrace you included below):
mvebu_pcie_scan_bus
pci_scan_root_bus
pci_create_root_bus
pci_scan_child_bus
pci_bus_add_devices
pci_bus_add_device
pci_fixup_device(pci_fixup_final)
quirk_usb_early_handoff # pci_fixup_final
quirk_usb_handoff_xhci
The problem is that we haven't assigned resources anywhere. Normally this is done by pci_bus_assign_resources() or pci_assign_unassigned_bus_resources(), but I don't think there's anything in the path above that does this.
This is not really a problem in your code; it's a generic PCI core problem. pci_scan_root_bus() does everything including creating the root bus, scanning it, and adding the devices we find. At the point where we add a device (pci_bus_add_device()), it should be ready for a driver to claim it -- all resource assignment should already be done.
I don't think it's completely trivial to fix this in the PCI core yet (but we're moving in that direction) because we have some boot-time ordering issues, e.g., x86 scans the root buses before we know about the address space consumed by ACPI devices, so we can't just assign the resources when we scan the bus.
I think the way you'll have to fix this in the meantime is to use pci_create_root_bus() directly so you can do things in this sequence:
pci_create_root_bus
pci_scan_child_bus
pci_bus_assign_resources
pci_bus_add_devices
dino_probe() is an example of similar code that does this.
Bjorn
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Bjorn Helgaas
I think the way you'll have to fix this in the meantime is to use
pci_create_root_bus()directly so you can do things in this sequence:
pci_create_root_bus
pci_scan_child_bus
pci_bus_assign_resources
pci_bus_add_devices
dino_probe()is an example of similar code that does this.
Erm, but isn't this problem going to strike any system that doesn't have resources assigned by the firmware?
If so, a common 'pci_scan_bus_from_scratch' would at least highlight the problem and centralize the work around until it is fixed...
Many embedded systems will require this..
Regards,
Jason
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 12 lines · Jason Gunthorpe
Quoted 9 lines · Bjorn Helgaas
I think the way you'll have to fix this in the meantime is to use
pci_create_root_bus()directly so you can do things in this sequence:
pci_create_root_bus
pci_scan_child_bus
pci_bus_assign_resources
pci_bus_add_devices
dino_probe()is an example of similar code that does this.
Quoted 7 lines
Erm, but isn't this problem going to strike any system that doesn't have resources assigned by the firmware?
If so, a common '
pci_scan_bus_from_scratch' would at least highlight the problem and centralize the work around until it is fixed...Many embedded systems will require this..
Yep, agreed on all counts.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Bjorn HelgaasQuoted 51 lines · Bjorn Helgaas
Quoted 11 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 4 lines · Bjorn Helgaas
Can you post the entire dmesg log, ideally with
CONFIG_PCI_DEBUG=y? That should have more information about the enumeration process, including what we think the XHCI BARs are and the apertures leading to them.Sure, see below.
Quoted 7 lines
Quoted 3 lines
The PCI core assumes that we know the host bridge apertures up front, and I'm not sure that is true on your platform, so maybe we'll need some changes to accommodate that.
In this hardware, we need to set up the address decoding windows. So there shouldn't be any access to a PCI device memory or I/O region until the addresses have been assigned in the PCI-to-PCI bridge.
I think this is the path where the crash happens (this is the same as the backtrace you included below):
mvebu_pcie_scan_bus
pci_scan_root_bus
pci_create_root_bus
pci_scan_child_bus
pci_bus_add_devices
pci_bus_add_device
pci_fixup_device(pci_fixup_final)
quirk_usb_early_handoff#pci_fixup_final
quirk_usb_handoff_xhciThe problem is that we haven't assigned resources anywhere. Normally this is done by
pci_bus_assign_resources()orpci_assign_unassigned_bus_resources(), but I don't think there's anything in the path above that does this.This is not really a problem in your code; it's a generic PCI core problem.
pci_scan_root_bus()does everything including creating the root bus, scanning it, and adding the devices we find. At the point where we add a device (pci_bus_add_device()), it should be ready for a driver to claim it -- all resource assignment should already be done.I don't think it's completely trivial to fix this in the PCI core yet (but we're moving in that direction) because we have some boot-time ordering issues, e.g., x86 scans the root buses before we know about the address space consumed by ACPI devices, so we can't just assign the resources when we scan the bus.
Quoted 7 lines
I think the way you'll have to fix this in the meantime is to use
pci_create_root_bus()directly so you can do things in this sequence:
pci_create_root_bus
pci_scan_child_bus
pci_bus_assign_resources
pci_bus_add_devices
The last two are already done by ARM's pci_common_init(). On Tegra I've got by having a custom .scan_bus() implementation that does the first two. Back when I used the pci_scan_root_bus() we were seeing some issues with resource conflicts and such and there was a similar discussion at that time.
Thomas, have you tried using the same .scan_bus() that I use on Tegra? It should be easy to port it to Marvell, the only Tegra-specific bit is how to get at the struct tegra_pcie.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130207/d1055f0b/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 14 lines · Thomas Petazzoni
Dear Stephen Warren,
Quoted 7 lines · Stephen Warren
Quoted 3 lines
+Mandatory properties:
+- compatible: must be "marvell,armada-370-xp-pcie" +- status: either "disabled" or "okay"status is a standard DT property; I certainly wouldn't expect its presence to be mandatory (there's a defined default), nor would I expect each device's binding to redefine this property.
Ok.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
+- marvell,pcie-port: the physical PCIe port number
Should the standardized cell-index property be used here instead? Or, perhaps that property is deprecated/discouraged...
The problem is that I need two identifiers, the pcie-port and pcie-lane, and it would be strange to have one referenced as cell-index, and the other one as marvell,pcie-lane, no?
Yes, using a custom property for half of the information and a standard property for the other half would be odd.
Quoted 6 lines
Unless of
course we can put two numbers in the cell-index property, but a quick grep inDocumentation/devicetree/bindings/seems to indicate that all users of cell-index use it with a single identifier.Just tell me what to do here, I don't have a strong opinion on this.
It's probably fine as-is then. Although I wasn't sure exactly what port/lane meant; is there some kind of mux/cross-bar between the PCIe root ports and the physical lanes/balls/pins on the chip?
Quoted 10 lines
Quoted 9 lines
drivers/pci/host/Makefile line 4
Quoted 6 lines
drivers/pci/host/Makefile +4
-0+obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +ccflags-$(CONFIG_PCI_MVEBU) += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/includeThat seems a little dangerous w.r.t. multi-platform zImage. Can the required headers be moved out to somewhere more public to avoid this?
Quoted 4 lines
Why is this dangerous for multi-platform zImage? For this specific driver only, some SoC-specific headers are used. I don't think it prevents another PCI driver (such as the Tegra one) from being built into the same kernel image, no?
Aren't those ccflags applied to everything that's built by that Makefile? If they were applied only to one .o file, it'd probably be OK, but I don't see how that's specified.
I'm not especially bothered with reaching into the mach/plat include directories especially since you're well aware it needs cleaning up, I just don't think that Tegra's PCIe driver is going to compile too well against an Orion/mvebu header if one was to get picked up first.
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 10 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the
hw_pcioperations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and thepci_opsoperations that are used by the PCI core to read/write the configuration space of PCI devices.
Quoted 4 lines
Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
There's no Linux requirement that multiple PCIe interfaces appear to be in the same hierarchy. You can just use pci_scan_root_bus() separately on each interface. Each interface can be in its own domain if necessary.
Quoted 290 lines
In addition, this driver enumerates the different PCIe slots, and for those having a device plugged in, it sets up the necessary address decoding windows, using the new
armada_370_xp_alloc_pcie_window()function frommach-mvebu/addr-map.c.Signed-off-by Thomas Petazzoni <[email protected]>--- .../devicetree/bindings/pci/armada-370-xp-pcie.txt | 175 +++++++ drivers/pci/host/Kconfig | 6 + drivers/pci/host/Makefile | 4 + drivers/pci/host/pci-mvebu.c | 500 ++++++++++++++++++++ 4 files changed, 685 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/armada-370-xp-pcie.txt create mode 100644 drivers/pci/host/Makefile create mode 100644 drivers/pci/host/pci-mvebu.cDocumentation/devicetree/bindings/pci/armada-370-xp-pcie.txt new file +175
-0@@ -0,0 +1,175 @@ +* Marvell Armada 370/XP PCIe interfaces + +Mandatory properties: +- compatible: must be "marvell,armada-370-xp-pcie" +- status: either "disabled" or "okay" +- #address-cells, set to <3> +- #size-cells, set to <2> +- #interrupt-cells, set to <1> +- bus-range: PCI bus numbers covered +- ranges: standard PCI-style address ranges, describing the PCIe + registers for each PCIe interface, and then ranges for the PCI + memory and I/O regions. +- interrupt-map-mask and interrupt-map are standard PCI Device Tree + properties to describe the interrupts associated to each PCI + interface. + +In addition, the Device Tree node must have sub-nodes describing each +PCIe interface, having the following mandatory properties: +- reg: the address and size of the PCIe registers (translated + addresses according to the ranges property of the parent) +- clocks: the clock associated to this PCIe interface +- marvell,pcie-port: the physical PCIe port number +- status: either "disabled" or "okay" + +and the following optional properties: +- marvell,pcie-lane: the physical PCIe lane number, for ports having + multiple lanes. If this property is not found, we assume that the + value is 0. + +Example: + +pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x00002800 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x00005000 0 0xd0082000 0xd0082000 0 0x00002000 /* port 3.0 registers */ + 0x00003000 0 0xd0084000 0xd0084000 0 0x00002000 /* port 1.1 registers */ + 0x00003800 0 0xd0088000 0xd0088000 0 0x00002000 /* port 1.2 registers */ + 0x00004000 0 0xd008C000 0xd008C000 0 0x00002000 /* port 1.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00100000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>; + interrupt-map = <0x0800 0 0 1 &mpic 58 + 0x1000 0 0 1 &mpic 59 + 0x1800 0 0 1 &mpic 60 + 0x2000 0 0 1 &mpic 61 + 0x2800 0 0 1 &mpic 62 + 0x3000 0 0 1 &mpic 63 + 0x3800 0 0 1 &mpic 64 + 0x4000 0 0 1 &mpic 65 + 0x4800 0 0 1 &mpic 99 + 0x5000 0 0 1 &mpic 103>; + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 0,1 { + device_type = "pciex"; + reg = <0x1000 0 0xd0044000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 6>; + status = "disabled"; + }; + + pcie at 0,2 { + device_type = "pciex"; + reg = <0x1800 0 0xd0048000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 7>; + status = "disabled"; + }; + + pcie at 0,3 { + device_type = "pciex"; + reg = <0x2000 0 0xd004C000 0 0xC000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 8>; + status = "disabled"; + }; + + pcie at 1,0 { + device_type = "pciex"; + reg = <0x2800 0 0xd0080000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 9>; + status = "disabled"; + }; + + pcie at 1,1 { + device_type = "pciex"; + reg = <0x3000 0 0xd0084000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <1>; + clocks = <&gateclk 10>; + status = "disabled"; + }; + + pcie at 1,2 { + device_type = "pciex"; + reg = <0x3800 0 0xd0088000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <2>; + clocks = <&gateclk 11>; + status = "disabled"; + }; + + pcie at 1,3 { + device_type = "pciex"; + reg = <0x4000 0 0xd008C000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <3>; + clocks = <&gateclk 12>; + status = "disabled"; + }; + pcie at 2,0 { + device_type = "pciex"; + reg = <0x4800 0 0xd0042000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <2>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 26>; + status = "disabled"; + }; + + pcie at 3,0 { + device_type = "pciex"; + reg = <0x5000 0 0xd0082000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <3>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 27>; + status = "disabled"; + }; +};drivers/pci/host/Kconfig +6
-0@@ -1,4 +1,10 @@ menu "PCI host controller drivers" depends on PCI +config PCI_MVEBU + bool "Marvell EBU PCIe controller" + depends on ARCH_MVEBU + select PCI_SW_HOST_BRIDGE + select PCI_SW_PCI_PCI_BRIDGE + endmenudrivers/pci/host/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o +ccflags-$(CONFIG_PCI_MVEBU) += \ + -I$(srctree)/arch/arm/plat-orion/include \ + -I$(srctree)/arch/arm/mach-mvebu/includedrivers/pci/host/pci-mvebu.c new file +62
-0@@ -0,0 +1,500 @@ +/* +* PCIe driver for Marvell Armada 370 and Armada XP SoCs +* +* This file is licensed under the terms of the GNU General Public +* License version 2. This program is licensed "as is" without any +* warranty of any kind, whether express or implied. +*/ + +#include <linux/kernel.h> +#include <linux/pci.h> +#include <linux/clk.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/platform_device.h> +#include <linux/of_address.h> +#include <linux/of_pci.h> +#include <linux/of_irq.h> +#include <linux/of_platform.h> +#include <plat/pcie.h> +#include <mach/addr-map.h> + +/* +* Those are the product IDs used for the emulated PCI Host bridge and +* emulated PCI-to-PCI bridges. They are temporary until we get +* official IDs assigned. +*/ +#define MARVELL_EMULATED_HOST_BRIDGE_ID 4141 +#define MARVELL_EMULATED_PCI_PCI_BRIDGE_ID 4242 + +struct mvebu_pcie_port ; + +/* Structure representing all PCIe interfaces */ +struct mvebu_pcie { +struct pci_sw_host_bridge bridge; +struct platform_device *pdev; +struct mvebu_pcie_port *ports; +struct resource io; +struct resource mem; +struct resource busn; +int nports; +}; + +/* Structure representing one PCIe interface */ +struct mvebu_pcie_port { +void __iomem *base; +spinlock_t conf_lock; +int haslink; +u32 port; +u32 lane; +int devfn; +struct clk *clk; +struct pci_sw_pci_bridge bridge; +struct device_node *dn; +}; + +static inline struct mvebu_pcie *sys_to_pcie (struct pci_sys_data *sys) +{ +return sys->private_data; +} + +/* PCI configuration space write function */ +static int mvebu_pcie_wr_conf(struct pci_bus *bus, u32 devfn,
drivers/pci/host/pci-mvebu.c line 72
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ int where, int size, u32 val) +{ + struct mvebu_pcie *pcie = sys_to_pcie(bus->sysdata); + +if (bus->number !=0 ) { +/* +* Accessing a real PCIe interface, where the Linux +* virtual bus number is equal to the hardware PCIe +* interface number + 1 +*/
This is really weird. It doesn't seem like a good idea to me, but I don't understand the whole architecture.
Quoted 170 lines
drivers/pci/host/pci-mvebu.c new file +170
-0+struct mvebu_pcie_port *port; +unsigned long flags; +int porti, ret; + + porti = bus->number -1 ; +if (porti >= pcie->nports) +return PCIBIOS_DEVICE_NOT_FOUND ; + + port = &pcie->ports[porti]; + +if (!port->haslink) +return PCIBIOS_DEVICE_NOT_FOUND ; + +if (PCI_SLOT (devfn) !=0 ) +return PCIBIOS_DEVICE_NOT_FOUND ; + +spin_lock_irqsave (&port->conf_lock, flags); + ret =orion_pcie_wr_conf_bus (port->base, bus->number -1 , +PCI_DEVFN (1 ,PCI_FUNC (devfn)), + where, size, val); +spin_unlock_irqrestore (&port->conf_lock, flags); + +return ret; + }else { +/* +* Accessing the emulated PCIe devices. In the first +* slot, the emulated host bridge, and in the next +* slots, the PCI-to-PCI bridges that correspond to +* each PCIe hardware interface +*/ +if (PCI_SLOT (devfn) ==0 &&PCI_FUNC (devfn) ==0 ) +return pci_sw_host_bridge_write (&pcie->bridge, where, + size, val); +else if (PCI_SLOT (devfn) >=1 && +PCI_SLOT (devfn) <= pcie->nports) { +struct mvebu_pcie_port *port; +int porti =PCI_SLOT (devfn) -1 ; + port = &pcie->ports[porti]; +return pci_sw_pci_bridge_write (&port->bridge, where, + size, val); + }else { +return PCIBIOS_DEVICE_NOT_FOUND ; + } + } + +return PCIBIOS_SUCCESSFUL ; +} + +/* PCI configuration space read function */ +static int mvebu_pcie_rd_conf (struct pci_bus *bus,u32 devfn,int where, +int size,u32 *val) +{ +struct mvebu_pcie *pcie =sys_to_pcie (bus->sysdata); + +if (bus->number !=0 ) { +/* +* Accessing a real PCIe interface, where the Linux +* virtual bus number is equal to the hardware PCIe +* interface number + 1 +*/ +struct mvebu_pcie_port *port; +unsigned long flags; +int porti, ret; + + porti = bus->number -1 ; +if (porti >= pcie->nports) { + *val =0xffffffff ; +return PCIBIOS_DEVICE_NOT_FOUND ; + } + + port = &pcie->ports[porti]; + +if (!port->haslink ||PCI_SLOT (devfn) !=0 ) { + *val =0xffffffff ; +return PCIBIOS_DEVICE_NOT_FOUND ; + } + +spin_lock_irqsave (&port->conf_lock, flags); + ret =orion_pcie_rd_conf_bus (port->base, bus->number -1 , +PCI_DEVFN (1 ,PCI_FUNC (devfn)), + where, size, val); +spin_unlock_irqrestore (&port->conf_lock, flags); + +return ret; + }else { +/* +* Accessing the emulated PCIe devices. In the first +* slot, the emulated host bridge, and in the next +* slots, the PCI-to-PCI bridges that correspond to +* each PCIe hardware interface +*/ +if (PCI_SLOT (devfn) ==0 &&PCI_FUNC (devfn) ==0 ) +return pci_sw_host_bridge_read (&pcie->bridge, where, + size, val); +else if (PCI_SLOT (devfn) >=1 && +PCI_SLOT (devfn) <= pcie->nports) { +struct mvebu_pcie_port *port; +int porti =PCI_SLOT (devfn) -1 ; + port = &pcie->ports[porti]; +return pci_sw_pci_bridge_read (&port->bridge, where, + size, val); + }else { + *val =0xffffffff ; +return PCIBIOS_DEVICE_NOT_FOUND ; + } + } +} + +static struct pci_ops mvebu_pcie_ops = { + .read = mvebu_pcie_rd_conf, + .write = mvebu_pcie_wr_conf, +}; + +static int __init mvebu_pcie_setup (int nr,struct pci_sys_data *sys) +{ +struct mvebu_pcie *pcie =sys_to_pcie (sys); +int i; + +pci_add_resource_offset (&sys->resources, &pcie->io, sys->io_offset); +pci_add_resource_offset (&sys->resources, &pcie->mem, sys->mem_offset); +pci_add_resource (&sys->resources, &pcie->busn); + +pci_ioremap_io (nr *SZ_64K , pcie->io.start); + +for (i =0 ; i < pcie->nports; i++) { +struct mvebu_pcie_port *port = &pcie->ports[i]; +orion_pcie_set_local_bus_nr (port->base, i); +orion_pcie_setup (port->base); + } + +return 1 ; +} + +static int __init mvebu_pcie_map_irq (const struct pci_dev *dev,u8 slot,u8 pin) +{ +struct mvebu_pcie *pcie =sys_to_pcie (dev->bus->sysdata); +struct mvebu_pcie_port *port; +struct of_irq oirq; +u32 laddr[3 ]; +int ret; +__be32 intspec; + +/* +* Ignore requests related to the emulated host bridge or the +* emulated pci-to-pci bridges +*/ +if (!dev->bus->number) +return -1 ; + + port = &pcie->ports[dev->bus->number -1 ]; + +/* +* Build an laddr array that describes the PCI device in a DT +* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 ); + laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + } + +return irq_create_of_mapping (oirq.controller, oirq.specifier, + oirq.size); +}
drivers/pci/host/pci-mvebu.c line 252
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ +/* +* For a given PCIe interface (represented by a mvebu_pcie_port +* structure), we read the PCI configuration space of the +* corresponding PCI-to-PCI bridge in order to find out which range of +* I/O addresses and memory addresses have been assigned to this PCIe +* interface. Using these informations, we set up the appropriate +* address decoding windows so that the physical address are actually +* resolved to the right PCIe interface. +*/
Are you inferring the host bridge apertures by using the resources assigned to devices under the bridge, i.e., taking the union of all the BARs and PCI-to-PCI bridge apertures of devices on the root bus? If so, it would be much better to learn the host bridge apertures via some non-PCI mechanism that corresponds to the actual hardware configuration. The config of devices below the host bridge gives you hints, of course, but usually they don't consume all the available space.
Quoted 251 lines
drivers/pci/host/pci-mvebu.c new file +248
-1+static int mvebu_pcie_window_config_port (struct mvebu_pcie *pcie, +struct mvebu_pcie_port *port) +{ +unsigned long iobase =0 ; +int ret; + +if (port->bridge.iolimit >= port->bridge.iobase) { +unsigned long iolimit =0xFFF | + ((port->bridge.iolimit &0xF0 ) <<8 ) | + (port->bridge.iolimitupper <<16 ); + iobase = ((port->bridge.iobase &0xF0 ) <<8 ) | + (port->bridge.iobaseupper <<16 ); + ret =armada_370_xp_alloc_pcie_window (port->port, port->lane, + iobase, iolimit-iobase, +IORESOURCE_IO ); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: could not alloc PCIe %d:%d window for I/O [0x%lx; 0x%lx]\n" , + __func__, port->port, port->lane, + iobase, iolimit); + goto out_io; + } + } + +if (port->bridge.memlimit >= port->bridge.membase) { +unsigned long membase = + ((port->bridge.membase &0xFFF0 ) <<16 ); +unsigned long memlimit = + ((port->bridge.memlimit &0xFFF0 ) <<16 ) |0xFFFFF ; + ret =armada_370_xp_alloc_pcie_window (port->port, port->lane, + membase, memlimit-membase, +IORESOURCE_MEM ); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: could not alloc PCIe %d:%d window for MEM [0x%lx; 0x%lx]\n" , + __func__, port->port, port->lane, + membase, memlimit); + goto out_mem; + } + } + +out_mem: +if (port->bridge.iolimit >= port->bridge.iobase) +armada_370_xp_free_pcie_window (iobase); +out_io: +return ret; +} + +/* +* Set up the address decoding windows for all PCIe interfaces. +*/ +static int mvebu_pcie_window_config (struct mvebu_pcie *pcie) +{ +int i, ret; + +for (i =0 ; i < pcie->nports; i++) { +struct mvebu_pcie_port *port = &pcie->ports[i]; +if (!port->haslink) +continue ; + + ret =mvebu_pcie_window_config_port (pcie, port); +if (ret) +return ret; + } + +return 0 ; +} + +static resource_size_t mvebu_pcie_align_resource (struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align) +{ +if (!(res->flags &IORESOURCE_IO )) +return start; + +/* +* The I/O regions must be 64K aligned, because the +* granularity of PCIe I/O address decoding windows is 64 K +*/ +return round_up (start,SZ_64K ); +} + +static int mvebu_pcie_enable (struct mvebu_pcie *pcie) +{ +struct hw_pci hw; + +memset (&hw,0 ,sizeof (hw)); + + hw.nr_controllers =1 ; + hw.private_data = (void **)&pcie; + hw.setup = mvebu_pcie_setup; + hw.map_irq = mvebu_pcie_map_irq; + hw.align_resource = mvebu_pcie_align_resource; + hw.ops = &mvebu_pcie_ops; + +pci_common_init (&hw); + +return mvebu_pcie_window_config (pcie); +} + +static int __init mvebu_pcie_probe (struct platform_device *pdev) +{ +struct mvebu_pcie *pcie; +struct device_node *np = pdev->dev.of_node; +struct device_node *child; +const __be32 *range =NULL ; +struct resource res; +int i, ret; + + pcie =devm_kzalloc (&pdev->dev,sizeof (struct mvebu_pcie ), +GFP_KERNEL ); +if (!pcie) +return -ENOMEM ; + + pcie->pdev = pdev; + +pci_sw_host_bridge_init (&pcie->bridge); + pcie->bridge.vendor =PCI_VENDOR_ID_MARVELL ; + pcie->bridge.device =MARVELL_EMULATED_HOST_BRIDGE_ID ; + +/* Get the I/O and memory ranges from DT */ +while ((range =of_pci_process_ranges (np, &res, range)) !=NULL ) { +if (resource_type (&res) ==IORESOURCE_IO ) { +memcpy (&pcie->io, &res,sizeof (res)); + pcie->io.name ="I/O" ; + } +if (resource_type (&res) ==IORESOURCE_MEM ) { +memcpy (&pcie->mem, &res,sizeof (res)); + pcie->mem.name ="MEM" ; + } + } + +/* Get the bus range */ + ret =of_pci_parse_bus_range (np, &pcie->busn); +if (ret) { +dev_err (&pdev->dev,"failed to parse bus-range property: %d\n" , + ret); +return ret; + } + +for_each_child_of_node (pdev->dev.of_node, child) { +if (!of_device_is_available (child)) +continue ; + pcie->nports++; + } + + pcie->ports =devm_kzalloc (&pdev->dev, pcie->nports * +sizeof (struct mvebu_pcie_port ), +GFP_KERNEL ); +if (!pcie->ports) +return -ENOMEM ; + + i =0 ; +for_each_child_of_node (pdev->dev.of_node, child) { +struct mvebu_pcie_port *port = &pcie->ports[i]; + +if (!of_device_is_available (child)) +continue ; + +if (of_property_read_u32 (child,"marvell,pcie-port" , + &port->port)) { +dev_warn (&pdev->dev, +"ignoring PCIe DT node, missing pcie-port property\n" ); +continue ; + } + +if (of_property_read_u32 (child,"marvell,pcie-lane" , + &port->lane)) + port->lane =0 ; + + port->devfn =of_pci_get_devfn (child); +if (port->devfn <0 ) +continue ; + + port->base =of_iomap (child,0 ); +if (!port->base) { +dev_err (&pdev->dev,"PCIe%d.%d: cannot map registers\n" , + port->port, port->lane); +continue ; + } + +if (orion_pcie_link_up (port->base)) { + port->haslink =1 ; +dev_info (&pdev->dev,"PCIe%d.%d: link up\n" , + port->port, port->lane); + }else { + port->haslink =0 ; +dev_info (&pdev->dev,"PCIe%d.%d: link down\n" , + port->port, port->lane); + } + + port->clk =of_clk_get_by_name (child,NULL ); +if (!port->clk) { +dev_err (&pdev->dev,"PCIe%d.%d: cannot get clock\n" , + port->port, port->lane); +iounmap (port->base); + port->haslink =0 ; +continue ; + } + + port->dn = child; + +clk_prepare_enable (port->clk); +spin_lock_init (&port->conf_lock); + +pci_sw_pci_bridge_init (&port->bridge); + port->bridge.vendor =PCI_VENDOR_ID_MARVELL ; + port->bridge.device =MARVELL_EMULATED_PCI_PCI_BRIDGE_ID ; + port->bridge.primary_bus =0 ; + port->bridge.secondary_bus =PCI_SLOT (port->devfn); + port->bridge.subordinate_bus =PCI_SLOT (port->devfn); + + i++; + } + +mvebu_pcie_enable (pcie); + +return 0 ; +} + +static const struct of_device_id mvebu_pcie_of_match_table[] = { + { .compatible ="marvell,armada-370-xp-pcie" , }, + {}, +}; +MODULE_DEVICE_TABLE (of, mvebu_pcie_of_match_table); + +static struct platform_driver mvebu_pcie_driver = { + .driver = { + .owner =THIS_MODULE , + .name ="mvebu-pcie" , + .of_match_table = +of_match_ptr (mvebu_pcie_of_match_table), + }, +}; + +static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +} + +subsys_initcall (mvebu_pcie_init); + +MODULE_AUTHOR ("Thomas Petazzoni <[email protected]>" ); +MODULE_DESCRIPTION ("Marvell EBU PCIe driver" ); +MODULE_LICENSE ("GPL" ); --1.7.9.5
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 17 lines · Bjorn Helgaas
Quoted 10 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the
hw_pcioperations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and thepci_opsoperations that are used by the PCI core to read/write the configuration space of PCI devices.Quoted 4 lines
Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
Quoted 4 lines
There's no Linux requirement that multiple PCIe interfaces appear to be in the same hierarchy. You can just use
pci_scan_root_bus()separately on each interface. Each interface can be in its own domain if necessary.
What you suggest is basically what the Marvell driver did originally, the probelm is that Linux requires a pre-assigned aperture for each PCI domain/root bus, and these new chips have so many PCI-E ports that they can exhaust the physical address space, and also a limited internal HW resource for setting address routing.
Thus they require resource allocation that is sensitive to the devices present downstream.
By far the simplest solution is to merge all the physical links into a single domain and rely on existing PCI resource allocation code to drive allocation of scarce physical address space and demand allocate the HW routing resource (specifically there are enough resources to accomidate MMIO only devices on every bus, but not enough to accomidate MMIO and IO on every bus).
Quoted 10 lines
Quoted 9 lines
+/*
+ * For a given PCIe interface (represented by amvebu_pcie_port
+ * structure), we read the PCI configuration space of the
+ * corresponding PCI-to-PCI bridge in order to find out which range of
+ * I/O addresses and memory addresses have been assigned to this PCIe
+ * interface. Using these informations, we set up the appropriate
+ * address decoding windows so that the physical address are actually
+ * resolved to the right PCIe interface.
+ */
Quoted 2 lines
Are you inferring the host bridge apertures by using the resources assigned to devices under the bridge, i.e., taking the union of all
The flow is different, a portion of physical address space is set aside for use by PCI-E (via DT) and that portion is specified in the struct resource's ultimately attached to the PCI domain for the bus scan. You could call that the 'host bridge aperture' though it doesn't reflect any HW configuration at all. The values come from the device tree.
During the bus scan the Linux core code splits up that contiguous space and assigns to the PCI-PCI bridges and devices under that domain.
Each physical PCI-E link on the chip is seen by Linux through the SW emulated PCI-PCI bridge attached to bus 0. When Linux configures the bridge windows it triggers this code here to copy that window information from the PCI config space into non-standard internal HW registers.
The purpose of the SW PCI-PCI bridge and this code here is to give the Linux PCI core control over the window (MMIO,IO,busnr) assigned to the PCI-E link.
This arrangement with PCI-PCI bridges controlling address routing is part of the PCI-E standard, in this instance Marvell did not implement the required config space in HW so the driver is working around that deficiency.
Other drivers, like tegra have a similar design, but their hardware does implement PCI-PCI bridge configuration space and does drive address decoding through the HW PCI-PCI window registers.
Having PCI-E links be bridges, not domains/root_bus's is in-line with the standard and works better with the Linux PCI resource allocator.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Bjorn, Jason,
Quoted 59 lines · Jason Gunthorpe
Quoted 4 lines
There's no Linux requirement that multiple PCIe interfaces appear to be in the same hierarchy. You can just use
pci_scan_root_bus()separately on each interface. Each interface can be in its own domain if necessary.What you suggest is basically what the Marvell driver did originally, the probelm is that Linux requires a pre-assigned aperture for each PCI domain/root bus, and these new chips have so many PCI-E ports that they can exhaust the physical address space, and also a limited internal HW resource for setting address routing.
Thus they require resource allocation that is sensitive to the devices present downstream.
By far the simplest solution is to merge all the physical links into a single domain and rely on existing PCI resource allocation code to drive allocation of scarce physical address space and demand allocate the HW routing resource (specifically there are enough resources to accomidate MMIO only devices on every bus, but not enough to accomidate MMIO and IO on every bus).
Quoted 10 lines
Quoted 9 lines
+/*
+ * For a given PCIe interface (represented by amvebu_pcie_port
+ * structure), we read the PCI configuration space of the
+ * corresponding PCI-to-PCI bridge in order to find out which range of
+ * I/O addresses and memory addresses have been assigned to this PCIe
+ * interface. Using these informations, we set up the appropriate
+ * address decoding windows so that the physical address are actually
+ * resolved to the right PCIe interface.
+ */Quoted 2 lines
Are you inferring the host bridge apertures by using the resources assigned to devices under the bridge, i.e., taking the union of all
The flow is different, a portion of physical address space is set aside for use by PCI-E (via DT) and that portion is specified in the struct resource's ultimately attached to the PCI domain for the bus scan. You could call that the 'host bridge aperture' though it doesn't reflect any HW configuration at all. The values come from the device tree.
During the bus scan the Linux core code splits up that contiguous space and assigns to the PCI-PCI bridges and devices under that domain.
Each physical PCI-E link on the chip is seen by Linux through the SW emulated PCI-PCI bridge attached to bus 0. When Linux configures the bridge windows it triggers this code here to copy that window information from the PCI config space into non-standard internal HW registers.
The purpose of the SW PCI-PCI bridge and this code here is to give the Linux PCI core control over the window (MMIO,IO,busnr) assigned to the PCI-E link.
This arrangement with PCI-PCI bridges controlling address routing is part of the PCI-E standard, in this instance Marvell did not implement the required config space in HW so the driver is working around that deficiency.
Quoted 6 lines
Other drivers, like tegra have a similar design, but their hardware does implement PCI-PCI bridge configuration space and does drive address decoding through the HW PCI-PCI window registers.
Having PCI-E links be bridges, not domains/root_bus's is in-line with the standard and works better with the Linux PCI resource allocator.
Thanks a lot Jason for this explanation, I couldn't have explained it as clearly as you did.
Bjorn, does Jason's reply answers your questions? Or do you need other details?
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeQuoted 52 lines · Jason Gunthorpe
Quoted 17 lines · Bjorn Helgaas
Quoted 10 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the
hw_pcioperations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and thepci_opsoperations that are used by the PCI core to read/write the configuration space of PCI devices.Quoted 4 lines
Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
Quoted 4 lines
There's no Linux requirement that multiple PCIe interfaces appear to be in the same hierarchy. You can just use
pci_scan_root_bus()separately on each interface. Each interface can be in its own domain if necessary.What you suggest is basically what the Marvell driver did originally, the probelm is that Linux requires a pre-assigned aperture for each PCI domain/root bus, and these new chips have so many PCI-E ports that they can exhaust the physical address space, and also a limited internal HW resource for setting address routing.
Thus they require resource allocation that is sensitive to the devices present downstream.
By far the simplest solution is to merge all the physical links into a single domain and rely on existing PCI resource allocation code to drive allocation of scarce physical address space and demand allocate the HW routing resource (specifically there are enough resources to accomidate MMIO only devices on every bus, but not enough to accomidate MMIO and IO on every bus).
Quoted 10 lines
Quoted 9 lines
+/*
+ * For a given PCIe interface (represented by amvebu_pcie_port
+ * structure), we read the PCI configuration space of the
+ * corresponding PCI-to-PCI bridge in order to find out which range of
+ * I/O addresses and memory addresses have been assigned to this PCIe
+ * interface. Using these informations, we set up the appropriate
+ * address decoding windows so that the physical address are actually
+ * resolved to the right PCIe interface.
+ */Quoted 2 lines
Are you inferring the host bridge apertures by using the resources assigned to devices under the bridge, i.e., taking the union of all
Quoted 6 lines
The flow is different, a portion of physical address space is set aside for use by PCI-E (via DT) and that portion is specified in the struct resource's ultimately attached to the PCI domain for the bus scan. You could call that the 'host bridge aperture' though it doesn't reflect any HW configuration at all. The values come from the device tree.
I think I would understand this better if we had a concrete example to talk about, say a dmesg log and corresponding lspci -v output.
As I understand it, the DT is a description of the hardware, so in that sense, the DT can't set aside physical address space. It can describe what the hardware does with the address space, and I assume that's what you mean. Maybe the hardware isn't configurable, e.g., it is hard-wired to route certain address ranges to PCIe?
Quoted 26 lines
During the bus scan the Linux core code splits up that contiguous space and assigns to the PCI-PCI bridges and devices under that domain.
Each physical PCI-E link on the chip is seen by Linux through the SW emulated PCI-PCI bridge attached to bus 0. When Linux configures the bridge windows it triggers this code here to copy that window information from the PCI config space into non-standard internal HW registers.
The purpose of the SW PCI-PCI bridge and this code here is to give the Linux PCI core control over the window (MMIO,IO,busnr) assigned to the PCI-E link.
This arrangement with PCI-PCI bridges controlling address routing is part of the PCI-E standard, in this instance Marvell did not implement the required config space in HW so the driver is working around that deficiency.
Other drivers, like tegra have a similar design, but their hardware does implement PCI-PCI bridge configuration space and does drive address decoding through the HW PCI-PCI window registers.
Having PCI-E links be bridges, not domains/root_bus's is in-line with the standard and works better with the Linux PCI resource allocator.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Bjorn Helgaas,
Quoted 2 lines · Bjorn Helgaas
I think I would understand this better if we had a concrete example to talk about, say a dmesg log and corresponding lspci -v output.
Please note that the cover letter of this patch series has the lspci -vvv output.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Bjorn HelgaasQuoted 5 lines · Bjorn Helgaas
As I understand it, the DT is a description of the hardware, so in that sense, the DT can't set aside physical address space. It can describe what the hardware does with the address space, and I assume that's what you mean. Maybe the hardware isn't configurable, e.g., it is hard-wired to route certain address ranges to PCIe?
The DT is largely a description of the hardware, but when it comes to addresses, particularly HW programmable addresess, there is an general expectation that the driver/bootloader will program HW address decoders to either match the addresses given in the DT, or to new values guided by the DT addresses.
In a real sense that means the DT also describes the physical address map the kernel should use.
In the PCI-E case the DT PCI-E HW description includes physical address ranges to use for the MMIO/IO/PREFETCH PCI-E interface windows and the driver is expected to program the internal HW address decoders based on those address ranges.
The catch is that the hardware decoders are on a link-by-link basis, not on a root-complex basis, so the programming can only take place once the Linux kernel has done PCI resource assignment.
So when I say set aside, I mean for instance, the PCI-E entry in DT has 128M of physical address space marked for PCI MMIO use. The kernel does PCI resource allocation and the HW decoders in each link will be set to claim some portion of the 128M - based on the MMIO windows programmed on the PCI-PCI root port bridges. The reamining part of the 128M is dead address space, not claimed by any hardware block at all.
Jason
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 26 lines · Jason Gunthorpe
Quoted 5 lines · Bjorn Helgaas
As I understand it, the DT is a description of the hardware, so in that sense, the DT can't set aside physical address space. It can describe what the hardware does with the address space, and I assume that's what you mean. Maybe the hardware isn't configurable, e.g., it is hard-wired to route certain address ranges to PCIe?
The DT is largely a description of the hardware, but when it comes to addresses, particularly HW programmable addresess, there is an general expectation that the driver/bootloader will program HW address decoders to either match the addresses given in the DT, or to new values guided by the DT addresses.
In a real sense that means the DT also describes the physical address map the kernel should use.
In the PCI-E case the DT PCI-E HW description includes physical address ranges to use for the MMIO/IO/PREFETCH PCI-E interface windows and the driver is expected to program the internal HW address decoders based on those address ranges.
The catch is that the hardware decoders are on a link-by-link basis, not on a root-complex basis, so the programming can only take place once the Linux kernel has done PCI resource assignment.
Quoted 6 lines
So when I say set aside, I mean for instance, the PCI-E entry in DT has 128M of physical address space marked for PCI MMIO use. The kernel does PCI resource allocation and the HW decoders in each link will be set to claim some portion of the 128M - based on the MMIO windows programmed on the PCI-PCI root port bridges. The reamining part of the 128M is dead address space, not claimed by any hardware block at all.
Thanks, this really helps get to the issue that the PCI core will care about. The root ports look like normal bridges, so the core assumes it can manage their windows as needed, as long as the windows stay inside the host bridge apertures that are logically upstream from the root ports.
In your example, it sounds like the 128M should be treated as the host bridge aperture. Is there any reason not to do that? It sounds like there's no place you can actually program that 128M region into the hardware, and you would just program pieces of that region as root port windows. But that should be OK from the core's perspective.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 7 lines · Bjorn Helgaas
Quoted 6 lines
So when I say set aside, I mean for instance, the PCI-E entry in DT has 128M of physical address space marked for PCI MMIO use. The kernel does PCI resource allocation and the HW decoders in each link will be set to claim some portion of the 128M - based on the MMIO windows programmed on the PCI-PCI root port bridges. The reamining part of the 128M is dead address space, not claimed by any hardware block at all.
Quoted 5 lines
Thanks, this really helps get to the issue that the PCI core will care about. The root ports look like normal bridges, so the core assumes it can manage their windows as needed, as long as the windows stay inside the host bridge apertures that are logically upstream from the root ports.
Yes, that is basically correct. This is what the PCI-E specification says the root complex/root port should look like and this is what some SOC hardware implements fully in hardware. The small wrinkle with Marvell is that the PCI-PCI bridge config space is created by the driver since the HW does not expose a standard config space.
Quoted 5 lines
In your example, it sounds like the 128M should be treated as the host bridge aperture. Is there any reason not to do that? It sounds like there's no place you can actually program that 128M region into the hardware, and you would just program pieces of that region as root port windows. But that should be OK from the core's perspective.
AFAIK this is already what Thomas's driver is doing..
Regards,
Jason
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Jason Gunthorpe
Quoted 7 lines · Bjorn Helgaas
Quoted 6 lines
So when I say set aside, I mean for instance, the PCI-E entry in DT has 128M of physical address space marked for PCI MMIO use. The kernel does PCI resource allocation and the HW decoders in each link will be set to claim some portion of the 128M - based on the MMIO windows programmed on the PCI-PCI root port bridges. The reamining part of the 128M is dead address space, not claimed by any hardware block at all.
Quoted 5 lines
Thanks, this really helps get to the issue that the PCI core will care about. The root ports look like normal bridges, so the core assumes it can manage their windows as needed, as long as the windows stay inside the host bridge apertures that are logically upstream from the root ports.
Quoted 5 lines
Yes, that is basically correct. This is what the PCI-E specification says the root complex/root port should look like and this is what some SOC hardware implements fully in hardware. The small wrinkle with Marvell is that the PCI-PCI bridge config space is created by the driver since the HW does not expose a standard config space.
Oh, so the actual *root port* itself doesn't conform to the spec? Wow, that's worse than I expected.
Then I guess you have emulate it and make sure its config space is complete enough and functional enough so that all the link management, power management, AER, etc., code in the core works as well as it would with a conforming device.
Quoted 10 lines
Quoted 5 lines
In your example, it sounds like the 128M should be treated as the host bridge aperture. Is there any reason not to do that? It sounds like there's no place you can actually program that 128M region into the hardware, and you would just program pieces of that region as root port windows. But that should be OK from the core's perspective.
AFAIK this is already what Thomas's driver is doing..
Regards,
Jason
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 22 lines · Bjorn Helgaas
Quoted 14 lines · Jason Gunthorpe
Quoted 7 lines · Bjorn Helgaas
Quoted 6 lines
So when I say set aside, I mean for instance, the PCI-E entry in DT has 128M of physical address space marked for PCI MMIO use. The kernel does PCI resource allocation and the HW decoders in each link will be set to claim some portion of the 128M - based on the MMIO windows programmed on the PCI-PCI root port bridges. The reamining part of the 128M is dead address space, not claimed by any hardware block at all.
Quoted 5 lines
Thanks, this really helps get to the issue that the PCI core will care about. The root ports look like normal bridges, so the core assumes it can manage their windows as needed, as long as the windows stay inside the host bridge apertures that are logically upstream from the root ports.
Quoted 5 lines
Yes, that is basically correct. This is what the PCI-E specification says the root complex/root port should look like and this is what some SOC hardware implements fully in hardware. The small wrinkle with Marvell is that the PCI-PCI bridge config space is created by the driver since the HW does not expose a standard config space.
Quoted 7 lines
Oh, so the actual *root port* itself doesn't conform to the spec? Wow, that's worse than I expected.
Then I guess you have emulate it and make sure its config space is complete enough and functional enough so that all the link management, power management, AER, etc., code in the core works as well as it would with a conforming device.
I'm not sure the existing emulation in these patches is sufficient. For example, pci_sw_pci_bridge_write() updates bridge->membase when we write to the window register, but I don't see anything that updates the actual hardware decoder. That might be done in mvebu_pcie_window_config_port() via armada_370_xp_alloc_pcie_window(), but that looks like it's only done once. If the PCI core updates a root port window later, I don't see where the hardware decoder will be updated.
Maybe you're counting on the window assignments to be static? The PCI core doesn't guarantee anything like that, though in the absence of hotplug I don't know any reason why it would change things.
I also forgot about the bus number munging in mvebu_pcie_rd_conf(). The PCI core can update the bridge secondary/subordinate registers. It looks like you don't support writing to them, and the read path (pci_sw_pci_bridge_read()) looks like it doesn't do any translation between the hardware and Linux bus numbers. I don't understand the system well enough to know if this is an issue.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Bjorn Helgaas
I'm not sure the existing emulation in these patches is sufficient. For example,
pci_sw_pci_bridge_write()updatesbridge->membasewhen we write to the window register, but I don't see anything that updates the actual hardware decoder. That might be done inmvebu_pcie_window_config_port()viaarmada_370_xp_alloc_pcie_window(), but that looks like it's only done once. If the PCI core updates a root port window later, I don't see where the hardware decoder will be updated.
Quoted 3 lines
Maybe you're counting on the window assignments to be static? The PCI core doesn't guarantee anything like that, though in the absence of hotplug I don't know any reason why it would change things.
Agree..
Thomas, I think you need to directly update the Marvell hardware registers when config writes are made to the SW bridge. If this means it is too hard/complex to keep the code general then I'd say make it part of the Marvell host driver.
Quoted 6 lines
I also forgot about the bus number munging in
mvebu_pcie_rd_conf(). The PCI core can update the bridge secondary/subordinate registers. It looks like you don't support writing to them, and the read path (pci_sw_pci_bridge_read()) looks like it doesn't do any translation between the hardware and Linux bus numbers. I don't understand the system well enough to know if this is an issue.
I was chatting with Thomas on this subject, it looks like there is a HW register that needs to be set to the subordinate bus number of the bridge, that will solve this weirdness.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Bjorn HelgaasDear Bjorn Helgaas,
Quoted 6 lines · Bjorn Helgaas
I'm not sure the existing emulation in these patches is sufficient. For example,
pci_sw_pci_bridge_write()updatesbridge->membasewhen we write to the window register, but I don't see anything that updates the actual hardware decoder. That might be done inmvebu_pcie_window_config_port()viaarmada_370_xp_alloc_pcie_window(), but that looks like it's only done once.
That's correct. I currently let the Linux PCI core enumerate the real PCIe devices, allocate the resources, and set the appropriate values in the emulated PCI-to-PCI bridge registers. Once this is all done, the Marvell PCIe driver looks at each PCI-to-PCI bridge, reads the membase and iobase registers, and creates address decoding windows so that the physical addresses assigned by the Linux PCI core actually resolve to the right PCIe interface. This is done once for all.
Quoted 2 lines
If the PCI core updates a root port window later, I don't see where the hardware decoder will be updated.
It will not be updated.
Quoted 3 lines
Maybe you're counting on the window assignments to be static? The PCI core doesn't guarantee anything like that, though in the absence of hotplug I don't know any reason why it would change things.
Right. Is supporting hotplug a show-stopper to get this included? I think it could be added later, if it happens to be needed, no?
I could of course do it, but the patch series is already quite large and complicated, so if we could merge a simple, but working, version first, and then improve on top of it when needed, it would be nice.
Quoted 6 lines
I also forgot about the bus number munging in
mvebu_pcie_rd_conf(). The PCI core can update the bridge secondary/subordinate registers. It looks like you don't support writing to them, and the read path (pci_sw_pci_bridge_read()) looks like it doesn't do any translation between the hardware and Linux bus numbers. I don't understand the system well enough to know if this is an issue.
Right. Could you explain a little bit for what reasons the PCI core could update the secondary/subordinate registers, and to what values it sets them?
For now, I statically assign the secondary bus register value to be X+1, where X is the number of the PCIe interface, since X=0 is reserved for the root bus (which has the host bridge and the PCI-to-PCI bridges).
Also, could you detail what kind of translation I should be doing when reading the hardware and Linux bus numbers?
I apologize for asking so many, probably silly, questions, but I am still learning all those internal PCI mechanisms.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 28 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 6 lines · Bjorn Helgaas
I'm not sure the existing emulation in these patches is sufficient. For example,
pci_sw_pci_bridge_write()updatesbridge->membasewhen we write to the window register, but I don't see anything that updates the actual hardware decoder. That might be done inmvebu_pcie_window_config_port()viaarmada_370_xp_alloc_pcie_window(), but that looks like it's only done once.That's correct. I currently let the Linux PCI core enumerate the real PCIe devices, allocate the resources, and set the appropriate values in the emulated PCI-to-PCI bridge registers. Once this is all done, the Marvell PCIe driver looks at each PCI-to-PCI bridge, reads the membase and iobase registers, and creates address decoding windows so that the physical addresses assigned by the Linux PCI core actually resolve to the right PCIe interface. This is done once for all.
Quoted 2 lines
If the PCI core updates a root port window later, I don't see where the hardware decoder will be updated.
It will not be updated.
Quoted 3 lines
Maybe you're counting on the window assignments to be static? The PCI core doesn't guarantee anything like that, though in the absence of hotplug I don't know any reason why it would change things.
Quoted 6 lines
Right. Is supporting hotplug a show-stopper to get this included? I think it could be added later, if it happens to be needed, no?
I could of course do it, but the patch series is already quite large and complicated, so if we could merge a simple, but working, version first, and then improve on top of it when needed, it would be nice.
I'm most concerned about the stuff in drivers/pci. I hesitate to merge drivers/pci/sw-pci-pci-bridge.c as-is because it's a model that's not connected to hardware and only works in a completely static situation, and the rest of the PCI core can't really deal with that.
But I don't think supporting hotplug should be a show-stopper at this point, either. It sounds like we might be heading towards hooking this up more directly to the Marvell hardware, which will make it more arch-dependent. Something like that could either go in arch/arm, or in some not-quite-so-generic spot under drivers/pci.
Quoted 10 lines
Quoted 6 lines
I also forgot about the bus number munging in
mvebu_pcie_rd_conf(). The PCI core can update the bridge secondary/subordinate registers. It looks like you don't support writing to them, and the read path (pci_sw_pci_bridge_read()) looks like it doesn't do any translation between the hardware and Linux bus numbers. I don't understand the system well enough to know if this is an issue.Right. Could you explain a little bit for what reasons the PCI core could update the secondary/subordinate registers, and to what values it sets them?
The secondary/subordinate registers effectively define a bus number aperture that tells the bridge which transactions to claim and forward downstream. When enumerating devices, we may update the subordinate bus number to widen the aperture so we can enumerate an arbitrary tree behind the bridge. When we're finished, we'll probably narrow it by updating the subordinate again, so the unused bus number space can be used for other bridges. I don't know the exact details of the algorithm, and they're likely to change anyway, but pci_scan_bridge() is where most of it happens.
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
Quoted 4 lines
For now, I statically assign the secondary bus register value to be X+1, where X is the number of the PCIe interface, since X=0 is reserved for the root bus (which has the host bridge and the PCI-to-PCI bridges).
That makes sense but limits you to a single bus (and really, a single device since this is PCIe) below the bridge.
Quoted 2 lines
Also, could you detail what kind of translation I should be doing when reading the hardware and Linux bus numbers?
I'm hoping that the register Jason mentioned is enough to avoid the need for translation. If it's not, we can explore this a bit more.
Bjorn
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Bjorn Helgaas,
Quoted 10 lines · Bjorn Helgaas
I'm most concerned about the stuff in
drivers/pci. I hesitate to mergedrivers/pci/sw-pci-pci-bridge.cas-is because it's a model that's not connected to hardware and only works in a completely static situation, and the rest of the PCI core can't really deal with that.But I don't think supporting hotplug should be a show-stopper at this point, either. It sounds like we might be heading towards hooking this up more directly to the Marvell hardware, which will make it more arch-dependent. Something like that could either go in
arch/arm, or in some not-quite-so-generic spot underdrivers/pci.
If you really don't want sw-pci-pci-bridge.c in drivers/pci, then I can make it a part of the drivers/pci/host/pci-mvebu.c driver itself. I initially followed the idea started by Thierry Redding for the emulated host bridge, but if you feel that this emulated PCI-to-PCI bridge is too specific to this driver, then I'm fine with keeping it inside the driver itself.
Quoted 21 lines
Quoted 10 lines
Quoted 6 lines
I also forgot about the bus number munging in
mvebu_pcie_rd_conf(). The PCI core can update the bridge secondary/subordinate registers. It looks like you don't support writing to them, and the read path (pci_sw_pci_bridge_read()) looks like it doesn't do any translation between the hardware and Linux bus numbers. I don't understand the system well enough to know if this is an issue.Right. Could you explain a little bit for what reasons the PCI core could update the secondary/subordinate registers, and to what values it sets them?
The secondary/subordinate registers effectively define a bus number aperture that tells the bridge which transactions to claim and forward downstream. When enumerating devices, we may update the subordinate bus number to widen the aperture so we can enumerate an arbitrary tree behind the bridge. When we're finished, we'll probably narrow it by updating the subordinate again, so the unused bus number space can be used for other bridges. I don't know the exact details of the algorithm, and they're likely to change anyway, but
pci_scan_bridge()is where most of it happens.
Quoted 3 lines
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
In practice, in our situation, there isn't a tree below the bridge. There is one single device. I'd prefer to not implement features that I cannot effectively test, and let the implementation of those additional features to whoever will need them, and therefore be able to test them.
I guess that if I integrate the PCI-to-PCI bridge emulation code within the Marvell driver, then I can keep it fairly limited to whatever the Marvell PCI driver requires, no?
Quoted 7 lines
Quoted 4 lines
For now, I statically assign the secondary bus register value to be X+1, where X is the number of the PCIe interface, since X=0 is reserved for the root bus (which has the host bridge and the PCI-to-PCI bridges).
That makes sense but limits you to a single bus (and really, a single device since this is PCIe) below the bridge.
Which is exactly what is happening here.
Quoted 5 lines
Quoted 2 lines
Also, could you detail what kind of translation I should be doing when reading the hardware and Linux bus numbers?
I'm hoping that the register Jason mentioned is enough to avoid the need for translation. If it's not, we can explore this a bit more.
Ok.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Thomas Petazzoni
Quoted 3 lines
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
In practice, in our situation, there isn't a tree below the bridge. There is one single device. I'd prefer to not implement features that I cannot effectively test, and let the implementation of those additional features to whoever will need them, and therefore be able to test them.
Agreed it is hard to test, but be aware that any system that has PCI-E slots can host an add-in card that has a bridge on it. These are midly common in some areas like high port count ethernet cards.
If you aren't going to attempt the implementation then a really big FIXME that the config access routing is not correct and needs to be based on the bus range assigned to the bridge would be friendly :)
Jason
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 4 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
On Wed, 30 Jan 2013 11:52:15 -0700, Bjorn Helgaas wrote:
Quoted 8 lines
Quoted 3 lines
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
In practice, in our situation, there isn't a tree below the bridge. There is one single device. I'd prefer to not implement features that I cannot effectively test, and let the implementation of those additional features to whoever will need them, and therefore be able to test them.
I understand the concern about testing, but my advice is to not use that as an excuse to artificially limit the functionality of the code you're writing :)
You're talking about emulating a bridge, and the bridge really doesn't know or care what's downstream. If it works with a single device downstream, it should work with another bridge downstream. Many aspects of bridge configuration can be tested with creative application of setpci, so that might be a possibility, too.
Quoted 3 lines
I guess that if I integrate the PCI-to-PCI bridge emulation code within the Marvell driver, then I can keep it fairly limited to whatever the Marvell PCI driver requires, no?
Yeah, it's just that the pci_dev for this emulated bridge is used by the generic PCI core code, e.g., pci_scan_bridge(), so whatever emulation you do has to be robust enough that the core won't notice it's emulated.
Bjorn
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Bjorn HelgaasDear Bjorn Helgaas,
Quoted 10 lines · Bjorn Helgaas
The secondary/subordinate registers effectively define a bus number aperture that tells the bridge which transactions to claim and forward downstream. When enumerating devices, we may update the subordinate bus number to widen the aperture so we can enumerate an arbitrary tree behind the bridge. When we're finished, we'll probably narrow it by updating the subordinate again, so the unused bus number space can be used for other bridges. I don't know the exact details of the algorithm, and they're likely to change anyway, but
pci_scan_bridge()is where most of it happens.
Quoted 3 lines
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
The PCI-to-PCI bridge specification says that the Primary Bus Number Register, Secondary Bus Number Register and Subordinate Bus Number Register of the PCI configuration space of a PCI-to-PCI bridge should all be set to 0 after reset.
Until now, I was forcing a specific value of the Secondary Bus Number and Subordinate Bus Number (1 for my first bridge, 2 for my second bridge, etc.).
Following you're recommendation, I've changed this, and left those values initialized to 0 by default, in order to let Linux set correct values. Yes, Linux does assign appropriate values in the Secondary Bus Number Register. But before that Linux also complains loudly that the bridge configuration is invalid:
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Looking at the code, we have:
/* Check if setup is sensible at all */
if (!pass &&
(primary != bus->number || secondary <= bus->number ||
secondary > subordinate)) {
dev_info(&dev->dev, "bridge configuration invalid ([bus %02x-%02x]), reconfiguring\n",
secondary, subordinate);
broken = 1;
}Due to the default values of the Primary Bus Number Register, Secondary Bus Number Register and Subordinate Bus Number Register, we have:
primary = 0
secondary = 0
subordinate = 0
We are enumerating the root bus, so bus->number = 0. Therefore:
* The test primary != bus->number is false, so it's not the problem.
* secondary <= bus->number is true, because secondary = 0, and bus->number = 0. It is the problem.
* secondary > subordinate is false.
So I'm not sure what to do with this...
Thoas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 33 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 10 lines · Bjorn Helgaas
The secondary/subordinate registers effectively define a bus number aperture that tells the bridge which transactions to claim and forward downstream. When enumerating devices, we may update the subordinate bus number to widen the aperture so we can enumerate an arbitrary tree behind the bridge. When we're finished, we'll probably narrow it by updating the subordinate again, so the unused bus number space can be used for other bridges. I don't know the exact details of the algorithm, and they're likely to change anyway, but
pci_scan_bridge()is where most of it happens.Quoted 3 lines
It looks like your current system doesn't support trees below the bridges, but hopefully we can make it so the generic enumeration algorithms still work.
The PCI-to-PCI bridge specification says that the Primary Bus Number Register, Secondary Bus Number Register and Subordinate Bus Number Register of the PCI configuration space of a PCI-to-PCI bridge should all be set to 0 after reset.
Until now, I was forcing a specific value of the Secondary Bus Number and Subordinate Bus Number (1 for my first bridge, 2 for my second bridge, etc.).
Following you're recommendation, I've changed this, and left those values initialized to 0 by default, in order to let Linux set correct values. Yes, Linux does assign appropriate values in the Secondary Bus Number Register. But before that Linux also complains loudly that the bridge configuration is invalid:
Quoted 6 lines
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Linux makes the unwarranted assumption that the PCI hierarchy has already been configured by firmware. If the only problem is the messages above, I think we could just rework the message so it doesn't look like an error. I would guess that we probably also see the same distressing message when we hot-add a card with a bridge on it, because firmware won't have initialized the bridge.
My rule of thumb is that I like to note something in dmesg about the initial configuration of bus/mem/io apertures and BARs, as well as indications when we update them. That way, the dmesg log should contain enough information to debug most enumeration and configuration defects. pci_scan_bridge() is somewhat lacking in this regard.
Bjorn
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Bjorn Helgaas,
Quoted 20 lines · Bjorn Helgaas
Quoted 6 lines
Following you're recommendation, I've changed this, and left those values initialized to 0 by default, in order to let Linux set correct values. Yes, Linux does assign appropriate values in the Secondary Bus Number Register. But before that Linux also complains loudly that the bridge configuration is invalid:
Quoted 6 lines
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Linux makes the unwarranted assumption that the PCI hierarchy has already been configured by firmware. If the only problem is the messages above, I think we could just rework the message so it doesn't look like an error. I would guess that we probably also see the same distressing message when we hot-add a card with a bridge on it, because firmware won't have initialized the bridge.
Quoted 5 lines
My rule of thumb is that I like to note something in dmesg about the initial configuration of bus/mem/io apertures and BARs, as well as indications when we update them. That way, the dmesg log should contain enough information to debug most enumeration and configuration defects.
pci_scan_bridge()is somewhat lacking in this regard.
Ok. Would something like:
"bridge configuration with unassigned bus numbers ([bus 00-00]), reconfiguring"
be an acceptable to replace this one?
Thanks for your quick feedback,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Bjorn Helgaas
maintainerPATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 30 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 20 lines · Bjorn Helgaas
Quoted 6 lines
Following you're recommendation, I've changed this, and left those values initialized to 0 by default, in order to let Linux set correct values. Yes, Linux does assign appropriate values in the Secondary Bus Number Register. But before that Linux also complains loudly that the bridge configuration is invalid:
Quoted 6 lines
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Linux makes the unwarranted assumption that the PCI hierarchy has already been configured by firmware. If the only problem is the messages above, I think we could just rework the message so it doesn't look like an error. I would guess that we probably also see the same distressing message when we hot-add a card with a bridge on it, because firmware won't have initialized the bridge.
Quoted 5 lines
My rule of thumb is that I like to note something in dmesg about the initial configuration of bus/mem/io apertures and BARs, as well as indications when we update them. That way, the dmesg log should contain enough information to debug most enumeration and configuration defects.
pci_scan_bridge()is somewhat lacking in this regard.
Quoted 5 lines
Ok. Would something like:
"bridge configuration with unassigned bus numbers ([bus 00-00]), reconfiguring"
be an acceptable to replace this one?
Seems reasonable.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Bjorn HelgaasQuoted 4 lines · Bjorn Helgaas
Linux makes the unwarranted assumption that the PCI hierarchy has already been configured by firmware. If the only problem is the messages above, I think we could just rework the message so it doesn't look like an error.
That's not a safe assumption, especially on platforms where there's no BIOS (like ARM platforms). Thankfully, for the platforms I care about, the boot loaders I wrote for them _do_ do a full bus setup, so I don't see a problem. :)
However, I have historially had the kernel over many years (probably around 14 or 15 now) reassign all resources so that things are how the kernel wants them, and not how my half-hearted attempt at setting them up did (which only does a limited job enough to get the system to a state where we can load the kernel.)
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 21 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
The driver implements the
hw_pcioperations needed by the core ARM PCI code to setup PCI devices and get their corresponding IRQs, and thepci_opsoperations that are used by the PCI core to read/write the configuration space of PCI devices.Since the PCIe interfaces of Marvell SoCs are completely separate and not linked together in a bus, this driver sets up an emulated PCI host bridge, with one PCI-to-PCI bridge as child for each hardware PCIe interface.
In addition, this driver enumerates the different PCIe slots, and for those having a device plugged in, it sets up the necessary address decoding windows, using the new
armada_370_xp_alloc_pcie_window()function frommach-mvebu/addr-map.c.Signed-off-by Thomas Petazzoni <[email protected]>
[snip]
Quoted 2 lines
(quoted diff) +2
-0+static int __init mvebu_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{
[snip]
Quoted 6 lines
drivers/pci/host/pci-mvebu.c new file +6
-0+ +/* +* Build an laddr array that describes the PCI device in a DT +* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );
drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }
Are you able to replace the above code with a call to of_irq_map_pci? I'm not sure which approach is better. The of_irq_map_pci function doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation of pcibios_get_phb_of_node for this to work correctly (see my RFC bios32 patch).
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +7
-0+ +return irq_create_of_mapping (oirq.controller, oirq.specifier, + oirq.size); +} +static int mvebu_pcie_enable (struct mvebu_pcie *pcie) +{ +struct hw_pci hw;
[snip]
Quoted 7 lines
drivers/pci/host/pci-mvebu.c new file +7
-0+pci_common_init (&hw); + +return mvebu_pcie_window_config (pcie); +} + +static int __init mvebu_pcie_probe (struct platform_device *pdev) +{
[snip]
Quoted 16 lines
drivers/pci/host/pci-mvebu.c new file +16
-0+ +mvebu_pcie_enable (pcie); + +return 0 ; +} + +static const struct of_device_id mvebu_pcie_of_match_table[] = { + { .compatible ="marvell,armada-370-xp-pcie" , }, + {}, +}; +MODULE_DEVICE_TABLE (of, mvebu_pcie_of_match_table); + +static struct platform_driver mvebu_pcie_driver = { + .driver = { + .owner =THIS_MODULE , + .name ="mvebu-pcie" ,
drivers/pci/host/pci-mvebu.c line 494
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ .of_match_table = + of_match_ptr(mvebu_pcie_of_match_table), + }, +}; + +static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +}
If you have multiple 'mvebu-pcie' in your DT then you will end up with multiple calls to mvebu_pcie_probe/mvebu_pcie_enable/pci_common_init.
However pci_common_init/pcibios_init_hw assumes it will only ever be called once, and will thus result in trying to create multiple busses with the same bus number. (The first root bus it creates is always zero provided you haven't implemented hw->scan).
I noticed this in Thierry's patch set and posted an RFC patch which overcomes this issue (patchwork.kernel.org/patch/2001171) and others. Perhaps you would want to include this in your patchset?
Andrew Murray
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Andrew Murray,
Quoted 22 lines · Andrew Murray
Quoted 2 lines
(quoted diff) +2
-0+static int __init mvebu_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{[snip]
Quoted 6 lines
drivers/pci/host/pci-mvebu.c new file +6
-0+ +/* +* Build an laddr array that describes the PCI device in a DT +* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }
Quoted 7 lines
Are you able to replace the above code with a call to
of_irq_map_pci? I'm not sure which approach is better. Theof_irq_map_pcifunction doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation ofpcibios_get_phb_of_nodefor this to work correctly (see my RFC bios32 patch).
I did try using the of_irq_map_pci() function, but unfortunately, it didn't work. IIRC, it didn't work because none of the pci_dev in my PCI tree had any 'struct device_node' associated to them, or at least not the one that had the right pdev->bus->number and pdev->devfn.
But, I guess that your patch that implements pcibios_get_phb_of_node() should fix this problem. I'll try this. Thanks!
Quoted 9 lines
drivers/pci/host/pci-mvebu.c line 494
Quoted 5 lines
drivers/pci/host/pci-mvebu.c new file +5
-0+static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +}If you have multiple 'mvebu-pcie' in your DT then you will end up with multiple calls to mvebu_pcie_probe/mvebu_pcie_enable/pci_common_init.
Right. In practice, there will only ever be a single DT node, since all PCIe interfaces are sub-nodes of the PCI controller node. But I understand the theoretical problem.
Quoted 8 lines
However pci_common_init/pcibios_init_hw assumes it will only ever be called once, and will thus result in trying to create multiple busses with the same bus number. (The first root bus it creates is always zero provided you haven't implemented
hw->scan).I noticed this in Thierry's patch set and posted an RFC patch which overcomes this issue (patchwork.kernel.org/patch/2001171) and others. Perhaps you would want to include this in your patchset?
Sure, I'll give it a test, and report if it works for me.
Thanks a lot!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 34 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 22 lines · Andrew Murray
Quoted 2 lines
(quoted diff) +2
-0+static int __init mvebu_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{[snip]
Quoted 6 lines
drivers/pci/host/pci-mvebu.c new file +6
-0+ +/* +* Build an laddr array that describes the PCI device in a DT +* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }Quoted 7 lines
Are you able to replace the above code with a call to
of_irq_map_pci? I'm not sure which approach is better. Theof_irq_map_pcifunction doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation ofpcibios_get_phb_of_nodefor this to work correctly (see my RFC bios32 patch).
Quoted 7 lines
I did try using the
of_irq_map_pci()function, but unfortunately, it didn't work. IIRC, it didn't work because none of thepci_devin my PCI tree had any 'structdevice_node' associated to them, or at least not the one that had the rightpdev->bus->numberandpdev->devfn.But, I guess that your patch that implements
pcibios_get_phb_of_node()should fix this problem. I'll try this. Thanks!
My bios32 patch departs slightly from your v2 04/27 patch in that it updates hw_pci to contain a device node rather than opaque private data and my pcibios_get_phb_of_node implementation relies on this. If you wanted to stick with the implementation you and Thierry share then you'd have to find another way to get to the device node from the void **private_data.
Quoted 34 lines
Quoted 9 lines
drivers/pci/host/pci-mvebu.c line 494
Quoted 5 lines
drivers/pci/host/pci-mvebu.c new file +5
-0+static int mvebu_pcie_init (void ) +{ +return platform_driver_probe (&mvebu_pcie_driver, + mvebu_pcie_probe); +}If you have multiple 'mvebu-pcie' in your DT then you will end up with multiple calls to mvebu_pcie_probe/mvebu_pcie_enable/pci_common_init.
Right. In practice, there will only ever be a single DT node, since all PCIe interfaces are sub-nodes of the PCI controller node. But I understand the theoretical problem.
Quoted 8 lines
However pci_common_init/pcibios_init_hw assumes it will only ever be called once, and will thus result in trying to create multiple busses with the same bus number. (The first root bus it creates is always zero provided you haven't implemented
hw->scan).I noticed this in Thierry's patch set and posted an RFC patch which overcomes this issue (patchwork.kernel.org/patch/2001171) and others. Perhaps you would want to include this in your patchset?
Sure, I'll give it a test, and report if it works for me.
Thanks a lot!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 40 lines · Andrew Murray
Quoted 39 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 22 lines · Andrew Murray
Quoted 2 lines
(quoted diff) +2
-0+static int __init mvebu_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{[snip]
Quoted 6 lines
drivers/pci/host/pci-mvebu.c new file +6
-0+ +/* +* Build an laddr array that describes the PCI device in a DT +* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }Quoted 7 lines
Are you able to replace the above code with a call to
of_irq_map_pci? I'm not sure which approach is better. Theof_irq_map_pcifunction doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation ofpcibios_get_phb_of_nodefor this to work correctly (see my RFC bios32 patch).I did try using the
of_irq_map_pci()function, but unfortunately, it didn't work. IIRC, it didn't work because none of thepci_devin my PCI tree had any 'structdevice_node' associated to them, or at least not the one that had the rightpdev->bus->numberandpdev->devfn.
Quoted 8 lines
Quoted 2 lines
But, I guess that your patch that implements
pcibios_get_phb_of_node()should fix this problem. I'll try this. Thanks!My bios32 patch departs slightly from your v2 04/27 patch in that it updates
hw_pcito contain a device node rather than opaque private data and mypcibios_get_phb_of_nodeimplementation relies on this. If you wanted to stick with the implementation you and Thierry share then you'd have to find another way to get to the device node from the void **private_data.
If at all possible I think the right thing to do is reuse the generic pcibios_get_phb_of_node() implementation. On Tegra this turned out to require a minimal change to the DT bindings of the root port nodes to make sure they provide the correct address in the reg property.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130129/47c1c5ce/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Thierry Reding,
Quoted 4 lines · Thierry Reding
If at all possible I think the right thing to do is reuse the generic
pcibios_get_phb_of_node()implementation. On Tegra this turned out to require a minimal change to the DT bindings of the root port nodes to make sure they provide the correct address in the reg property.
Could you detail the change that was needed? The DT bindings I use for the Marvell PCIe driver are very, very similar to the ones you use for Tegra, since I basically inspired my entire DT binding on your work. And still, I think of_irq_map_pci() wasn't working for me.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Thomas Petazzoni
Dear Thierry Reding,
Quoted 4 lines · Thierry Reding
If at all possible I think the right thing to do is reuse the generic
pcibios_get_phb_of_node()implementation. On Tegra this turned out to require a minimal change to the DT bindings of the root port nodes to make sure they provide the correct address in the reg property.
Quoted 4 lines
Could you detail the change that was needed? The DT bindings I use for the Marvell PCIe driver are very, very similar to the ones you use for Tegra, since I basically inspired my entire DT binding on your work. And still, I think
of_irq_map_pci()wasn't working for me.
Now that I think about it, there were a few more changes needed. For one, the reg property of the root port nodes need to be in the format specified by the PCI DT binding. That is, 3 cells for the address and 2 cells for the size.
So I end up with something like this:
pcie-controller {
...
ranges = <0x00000800 0 0x80000000 0x80000000 0 0x00001000 /* port 0 registers */
0x00001000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */
...>;
pci at 1,0 {
reg = <0x000800 0 0x80000000 0 0x1000>;
...
};
pci at 2,0 {
reg = <0x001000 0 0x80001000 0 0x1000>;
...
};
};So what happens here is that for each root port (pci at 1,0 and pci at 2,0), the reg property is translated into the parent address space via the pcie-controller's ranges property. pci at 1,0 gets the memory region 0x80000000-0x80000fff and pci at 2,0 gets 0x80001000-0x80001fff. (These are actually windows through which the configuration space of the root ports is accessed.)
At the same time this reg property maps both devices into the PCI address space at addresses 0:01.0 and 0:02.0 respectively.
The second change is that you can't rely on ARM's default implementation of the bus scan operation, which calls pci_scan_root_bus(), passing in a NULL as the struct device which acts as the bus' parent. So on Tegra I added a custom implementation which calls pci_create_root_bus(), passing in the struct device of the PCI host bridge, whose .of_node field will be set to the pcie-controller node above. Incidentally this also fixed another issue where the PCI core and ARM's pci_common_init() both eventually end up calling pci_bus_add_devices(). I don't remember the exact symptoms but I think this was causing resource conflicts during the second enumeration or so.
Because a proper struct device with the correct .of_node field is passed into pci_create_root_bus(), the generic pcibios_get_phb_of_node() will know how to find it by looking at bus->bridge->parent->of_node. After that the generic matching code will search the bridge (pcie-controller) node's children and relate them to the struct pci_dev by devfn. This is done in pci_set_of_node() defined in drivers/pci/of.c, which calls of_pci_find_child_device() from drivers/of/of_pci.c.
This is quite convoluted, but I hope it helps.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130129/1a0b14a8/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 69 lines · Thierry Reding
Quoted 9 lines · Thomas Petazzoni
Dear Thierry Reding,
Quoted 4 lines · Thierry Reding
If at all possible I think the right thing to do is reuse the generic
pcibios_get_phb_of_node()implementation. On Tegra this turned out to require a minimal change to the DT bindings of the root port nodes to make sure they provide the correct address in the reg property.Quoted 4 lines
Could you detail the change that was needed? The DT bindings I use for the Marvell PCIe driver are very, very similar to the ones you use for Tegra, since I basically inspired my entire DT binding on your work. And still, I think
of_irq_map_pci()wasn't working for me.Now that I think about it, there were a few more changes needed. For one, the reg property of the root port nodes need to be in the format specified by the PCI DT binding. That is, 3 cells for the address and 2 cells for the size.
So I end up with something like this:
pcie-controller { ... ranges = <0x00000800 0 0x80000000 0x80000000 0 0x00001000 /* port 0 registers */ 0x00001000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ ...>; pci at 1,0 { reg = <0x000800 0 0x80000000 0 0x1000>; ... }; pci at 2,0 { reg = <0x001000 0 0x80001000 0 0x1000>; ... }; };So what happens here is that for each root port (pci at 1,0 and pci at 2,0), the reg property is translated into the parent address space via the pcie-controller's ranges property. pci at 1,0 gets the memory region 0x80000000-0x80000fff and pci at 2,0 gets 0x80001000-0x80001fff. (These are actually windows through which the configuration space of the root ports is accessed.)
At the same time this reg property maps both devices into the PCI address space at addresses 0:01.0 and 0:02.0 respectively.
The second change is that you can't rely on ARM's default implementation of the bus scan operation, which calls
pci_scan_root_bus(), passing in a NULL as the struct device which acts as the bus' parent. So on Tegra I added a custom implementation which callspci_create_root_bus(), passing in the struct device of the PCI host bridge, whose .of_node field will be set to the pcie-controller node above. Incidentally this also fixed another issue where the PCI core and ARM'spci_common_init()both eventually end up callingpci_bus_add_devices(). I don't remember the exact symptoms but I think this was causing resource conflicts during the second enumeration or so.Because a proper struct device with the correct .of_node field is passed into
pci_create_root_bus(), the genericpcibios_get_phb_of_node()will know how to find it by looking atbus->bridge->parent->of_node. After that the generic matching code will search the bridge (pcie-controller) node's children and relate them to the structpci_devby devfn. This is done inpci_set_of_node()defined indrivers/pci/of.c, which callsof_pci_find_child_device()fromdrivers/of/of_pci.c.
Quoted 1 line
This is quite convoluted, but I hope it helps.
Thanks this is very helpful. I will see if this lets me avoid implementing pcibios_get_phb_of_node.
Thanks,
Andrew Murray
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingDear Thierry Reding,
Quoted 32 lines · Thierry Reding
Now that I think about it, there were a few more changes needed. For one, the reg property of the root port nodes need to be in the format specified by the PCI DT binding. That is, 3 cells for the address and 2 cells for the size.
So I end up with something like this:
pcie-controller { ... ranges = <0x00000800 0 0x80000000 0x80000000 0 0x00001000 /* port 0 registers */ 0x00001000 0 0x80001000 0x80001000 0 0x00001000 /* port 1 registers */ ...>; pci at 1,0 { reg = <0x000800 0 0x80000000 0 0x1000>; ... }; pci at 2,0 { reg = <0x001000 0 0x80001000 0 0x1000>; ... }; };So what happens here is that for each root port (pci at 1,0 and pci at 2,0), the reg property is translated into the parent address space via the pcie-controller's ranges property. pci at 1,0 gets the memory region 0x80000000-0x80000fff and pci at 2,0 gets 0x80001000-0x80001fff. (These are actually windows through which the configuration space of the root ports is accessed.)
Quoted 2 lines
At the same time this reg property maps both devices into the PCI address space at addresses 0:01.0 and 0:02.0 respectively.
This part I think I've done exactly the same thing in the Marvell PCIe DT binding.
Quoted 11 lines
The second change is that you can't rely on ARM's default implementation of the bus scan operation, which calls
pci_scan_root_bus(), passing in a NULL as the struct device which acts as the bus' parent. So on Tegra I added a custom implementation which callspci_create_root_bus(), passing in the struct device of the PCI host bridge, whose .of_node field will be set to the pcie-controller node above. Incidentally this also fixed another issue where the PCI core and ARM'spci_common_init()both eventually end up callingpci_bus_add_devices(). I don't remember the exact symptoms but I think this was causing resource conflicts during the second enumeration or so.
Quoted 7 lines
Because a proper struct device with the correct .of_node field is passed into
pci_create_root_bus(), the genericpcibios_get_phb_of_node()will know how to find it by looking atbus->bridge->parent->of_node. After that the generic matching code will search the bridge (pcie-controller) node's children and relate them to the structpci_devby devfn. This is done inpci_set_of_node()defined indrivers/pci/of.c, which callsof_pci_find_child_device()fromdrivers/of/of_pci.c.
This is quite certainly the part that I was missing. I'll try this and let you know.
Thanks a lot for the lengthy, but very useful explanation!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Andrew MurrayDear Andrew Murray,
Quoted 17 lines · Andrew Murray
Quoted 6 lines
(quoted diff) +2
-0+ /* + * Build an laddr array that describes the PCI device in aDT
drivers/pci/host/pci-mvebu.c new file +3
-0+* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }
Quoted 9 lines
Are you able to replace the above code with a call to
of_irq_map_pci? I'm not sure which approach is better. Theof_irq_map_pcifunction doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation ofpcibios_get_phb_of_nodefor this to work correctly (see my RFC bios32 patch).
I tried to do so, but it doesn't work properly. Let me explain what I did and the behavior that I observe.
First of all, I didn't reimplement the pcibios_get_phb_of_node(), but instead, as Thierry Reding suggested, simply implemented the hw_pci.scan() function as follows:
static struct pci_bus *mvebu_pcie_scan_bus(int nr, struct pci_sys_data *sys)
{
struct mvebu_pcie *pcie = sys_to_pcie(sys);
return pci_scan_root_bus(&pcie->pdev->dev, sys->busnr,
&mvebu_pcie_ops, sys, &sys->resources);
}This allows to pass the "struct device *" pointer, which ultimately allows the PCI devices to carry a pointer to the corresponding DT node.
The DT representing my PCIe controller and its interfaces is the following:
pcie-controller {
compatible = "marvell,armada-370-xp-pcie";
status = "disabled";
#address-cells = <3>;
#size-cells = <2>;
bus-range = <0x00 0xff>;
ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */
0x00001000 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */
0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */
0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */
#interrupt-cells = <1>;
interrupt-map-mask = <0xf800 0 0 1>;
interrupt-map = <0x0800 0 0 1 &mpic 58 /* port 0.0 */
0x1000 0 0 1 &mpic 62>; /* port 1.0 */
pcie at 0,0 {
device_type = "pciex";
reg = <0x0800 0 0xd0040000 0 0x2000>;
#address-cells = <3>;
#size-cells = <2>;
marvell,pcie-port = <0>;
marvell,pcie-lane = <0>;
clocks = <&gateclk 5>;
status = "disabled";
};
pcie at 1,0 {
device_type = "pciex";
reg = <0x1000 0 0xd0080000 0 0x2000>;
#address-cells = <3>;
#size-cells = <2>;
marvell,pcie-port = <1>;
marvell,pcie-lane = <0>;
clocks = <&gateclk 9>;
status = "disabled";
};
};So we have two PCIe interfaces. lspci shows the following output:
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 01:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300 02:00.0 USB controller: ASMedia Technology Inc. Device 1040
So:
* On bus 0, we have two PCI-to-PCI bridges. Those are emulated in software and allow the Marvell PCIe driver to get dynamic assignment of address decoding windows. The entries in the interrupt-map DT property match the bus number and slot number of those PCI-to-PCI bridges.
* On bus 1, we have the real PCIe device connected to the first PCIe interface. This bus 1 is made "visible" thanks to the 00:01.0 PCI-to-PCI bridge.
* On bus 2, we have the real PCIe device connected to the second PCIe interface. This bus 2 is made "visible" thanks to the 00:02.0 PCI-to-PCI bridge.
Now, when I call the of_irq_map_pci() function, the problem is that the "struct pci_dev" that it receives is the one corresponding to the particular PCIe device we're interested in. And this "struct pci_dev" has a non-NULL pointer to the "struct device_node" representing "pcie0,0" or "pci0,1" above. Since the "struct device_node" is non-NULL, of_irq_map_pci() builds a laddr[] with the bus number and devfn of this device: bus number is 1, devfn is 0. And this doesn't match with the interrupt-map that is in my DT, which associates the interrupts numbers with the PCI-to-PCI bridges rather than the devices themselves.
To me, the "struct pci_dev" representing the real PCIe devices should have a NULL "struct device_node" pointer, because those device are not represented in the DT. If this was the case, then the of_irq_map_pci() would go one level up in the PCI hierarchy, find the "struct pci_dev" that corresponds to the PCI-to-PCI bridge, which generates an laddr[] having a bus number a devfn value matching the interrupt-map property.
If I do the following hack in of_irq_map_pci(), then everything works nicely:
} else {
/* We found a P2P bridge, check if it has a node */
ppnode = pci_device_to_OF_node(ppdev);
+ if (pdev->bus->number != 0)
+ ppnode = NULL;
}What this hack does is that if we are not on bus 0, it means that the pci_dev is a real PCIe device, and therefore we force the code to asssume it does not have a DT reference.
Isn't there a problem here in the PCI/DT code ? Is it normal that a PCIe device that isn't described in the DT carries a non-NULL struct device_node pointer?
If you need some more details about the problem, do not hesitate to ask.
Thanks a lot for your help,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 31 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 17 lines · Andrew Murray
Quoted 6 lines
(quoted diff) +2
-0+ /* + * Build an laddr array that describes the PCI device in aDT
drivers/pci/host/pci-mvebu.c new file +3
-0+* way +*/ + laddr[0 ] =cpu_to_be32 (port->devfn <<8 );drivers/pci/host/pci-mvebu.c line 238
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+ laddr[1 ] = laddr[2 ] =0 ; + intspec =cpu_to_be32 (pin); + + ret =of_irq_map_raw (port->dn, &intspec,1 , laddr, &oirq); +if (ret) { +dev_err (&pcie->pdev->dev, +"%s: of_irq_map_raw() failed, %d\n" , + __func__, ret); +return ret; + }Quoted 9 lines
Are you able to replace the above code with a call to
of_irq_map_pci? I'm not sure which approach is better. Theof_irq_map_pcifunction doesn't require the pin argument and instead uses the DT and/or performs its own pin swizzling. I guess this means that if there are PCIe devices in the DT tree that does any thing strange with pins then it would be reflected in the IRQ you get. I've found that you will also need to provide an implementation ofpcibios_get_phb_of_nodefor this to work correctly (see my RFC bios32 patch).
Quoted 6 lines
I tried to do so, but it doesn't work properly. Let me explain what I did and the behavior that I observe.
First of all, I didn't reimplement the
pcibios_get_phb_of_node(), but instead, as Thierry Reding suggested, simply implemented thehw_pci.scan()function as follows:
I've not had any time to test Thierry's solution to avoid implementing pcibios_get_phb_of_node - but it did seem to work for him and seem correct at the time.
Quoted 90 lines
static struct
pci_bus*mvebu_pcie_scan_bus(int nr, structpci_sys_data*sys){ struct mvebu_pcie *pcie = sys_to_pcie(sys); return pci_scan_root_bus(&pcie->pdev->dev, sys->busnr, &mvebu_pcie_ops, sys, &sys->resources); }This allows to pass the "struct device *" pointer, which ultimately allows the PCI devices to carry a pointer to the corresponding DT node.
The DT representing my PCIe controller and its interfaces is the following:
pcie-controller { compatible = "marvell,armada-370-xp-pcie"; status = "disabled"; #address-cells = <3>; #size-cells = <2>; bus-range = <0x00 0xff>; ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ 0x00001000 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ #interrupt-cells = <1>; interrupt-map-mask = <0xf800 0 0 1>; interrupt-map = <0x0800 0 0 1 &mpic 58 /* port 0.0 */ 0x1000 0 0 1 &mpic 62>; /* port 1.0 */ pcie at 0,0 { device_type = "pciex"; reg = <0x0800 0 0xd0040000 0 0x2000>; #address-cells = <3>; #size-cells = <2>; marvell,pcie-port = <0>; marvell,pcie-lane = <0>; clocks = <&gateclk 5>; status = "disabled"; }; pcie at 1,0 { device_type = "pciex"; reg = <0x1000 0 0xd0080000 0 0x2000>; #address-cells = <3>; #size-cells = <2>; marvell,pcie-port = <1>; marvell,pcie-lane = <0>; clocks = <&gateclk 9>; status = "disabled"; }; };So we have two PCIe interfaces. lspci shows the following output:
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 01:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300 02:00.0 USB controller: ASMedia Technology Inc. Device 1040
So:
* On bus 0, we have two PCI-to-PCI bridges. Those are emulated in software and allow the Marvell PCIe driver to get dynamic assignment of address decoding windows. The entries in the interrupt-map DT property match the bus number and slot number of those PCI-to-PCI bridges.
* On bus 1, we have the real PCIe device connected to the first PCIe interface. This bus 1 is made "visible" thanks to the 00:01.0 PCI-to-PCI bridge.
* On bus 2, we have the real PCIe device connected to the second PCIe interface. This bus 2 is made "visible" thanks to the 00:02.0 PCI-to-PCI bridge.
Now, when I call the
of_irq_map_pci()function, the problem is that the "structpci_dev" that it receives is the one corresponding to the particular PCIe device we're interested in. And this "structpci_dev" has a non-NULL pointer to the "structdevice_node" representing "pcie0,0" or "pci0,1" above. Since the "structdevice_node" is non-NULL,of_irq_map_pci()builds a laddr[] with the bus number and devfn of this device: bus number is 1, devfn is 0. And this doesn't match with the interrupt-map that is in my DT, which associates the interrupts numbers with the PCI-to-PCI bridges rather than the devices themselves.
Quoted 6 lines
To me, the "struct
pci_dev" representing the real PCIe devices should have a NULL "structdevice_node" pointer, because those device are not represented in the DT. If this was the case, then theof_irq_map_pci()would go one level up in the PCI hierarchy, find the "structpci_dev" that corresponds to the PCI-to-PCI bridge, which generates an laddr[] having a bus number a devfn value matching the interrupt-map property.
Yes this is my current understanding.
Quoted 12 lines
If I do the following hack in
of_irq_map_pci(), then everything works nicely:} else { /* We found a P2P bridge, check if it has a node */ ppnode = pci_device_to_OF_node(ppdev);(quoted diff) +2
-0+ if (pdev->bus->number != 0) + ppnode = NULL; }
Quoted 7 lines
What this hack does is that if we are not on bus 0, it means that the
pci_devis a real PCIe device, and therefore we force the code to asssume it does not have a DT reference.Isn't there a problem here in the PCI/DT code ? Is it normal that a PCIe device that isn't described in the DT carries a non-NULL struct
device_nodepointer?
I would suggest the issue isn't in the PCI/DT code. This is what I see with my implementation (which uses an implementation of pcibios_get_phb_of_node) - I'd like to believe this is correct behaviour:
- of_irq_map_pci is called for an endpoint (5:0:0), its pdev->dev.of_node is NULL. As I don't have a representation of this endpoint in my DT the of_irq_map_pci code proceeds to walk the fabric.
- Starting with the pdev for 5:0:0, of_irq_map_pci sees it has a parent (downstream switch bridge), in this case pdev(5:0:0)->bus->self->dev.of_node is NULL, due to this swizzling occurs and we walk the parent's bus (bus 2)
- This continues to bus 1 and then bus 0 where of_irq_map_pci realises that it has no parent (its the host bridge). Due to the implementation of pcibios_get_phb_of_node a of_node is produced, this is then used to construct a lspec (for bus number 0).
The of_irq_map_pci code stops walking as soon as it finds a function in the tree that has a device node. This suggests that if you represent a bridge you must also include an interrupt-map. The problem here is that you have represented a bridge but not included a map.
I can think of three solutions:
1. Something like this:
} else {
/* We found a P2P bridge, check if it has a node */
ppnode = pci_device_to_OF_node(ppdev);
+ if (ppnode doesnt have an interrupt-map)//if (pdev->bus->number != 0)
+ ppnode = NULL;
}2. Remove the bridges from the DT? Or remove the map from pcie-controller and add a map each to pcie at 0,1 and pcie at 1,1?
3. Change the mask of your map so that it doesn't care about bus numbers. I have a map that looks like this:
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 ... //anything coming in on INTA
0 0 0 2 ... //anything coming in on INTB
0 0 0 3 ... ...
0 0 0 4 ... ...Andrew Murray
Quoted 12 lines
If you need some more details about the problem, do not hesitate to ask.
Thanks a lot for your help,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 3 lines
First of all, I didn't reimplement the
pcibios_get_phb_of_node(), but instead, as Thierry Reding suggested, simply implemented thehw_pci.scan()function as follows:I've not had any time to test Thierry's solution to avoid implementing
pcibios_get_phb_of_node- but it did seem to work for him and seem correct at the time.
At least there are device tree nodes associated to PCI devices, so it looks good from this perspective.
Quoted 8 lines
Quoted 6 lines
To me, the "struct
pci_dev" representing the real PCIe devices should have a NULL "structdevice_node" pointer, because those device are not represented in the DT. If this was the case, then theof_irq_map_pci()would go one level up in the PCI hierarchy, find the "structpci_dev" that corresponds to the PCI-to-PCI bridge, which generates an laddr[] having a bus number a devfn value matching the interrupt-map property.Yes this is my current understanding.
Ok. But that's not what happens: the "struct pci_dev" representing the real PCIe device *DOES* have a non-NULL struct device_node pointer. And this is what makes the entire thing fail.
Quoted 1 line
I would suggest the issue isn't in the PCI/DT code. This is what I see
I believe it is, because as I said above, the struct pci_dev associated to a real PCIe device should not have a struct device_node pointer, because this PCIe device has been dynamically enumerated and is therefore not part of the device tree.
Quoted 6 lines
with my implementation (which uses an implementation of
pcibios_get_phb_of_node) - I'd like to believe this is correct behaviour:-
of_irq_map_pciis called for an endpoint (5:0:0), itspdev->dev.of_nodeis NULL. As I don't have a representation of this endpoint in my DT theof_irq_map_pcicode proceeds to walk the fabric.
I believe this should be the behavior. But this not what happens: the pdev->dev.of_node of an endpoint pci_dev is not NULL.
Quoted 8 lines
- Starting with the pdev for 5:0:0,
of_irq_map_pcisees it has a parent (downstream switch bridge), in this case pdev(5:0:0)->bus->self->dev.of_node is NULL, due to this swizzling occurs and we walk the parent's bus (bus 2)
- This continues to bus 1 and then bus 0 whereof_irq_map_pcirealises that it has no parent (its the host bridge). Due to the implementation ofpcibios_get_phb_of_nodeaof_nodeis produced, this is then used to construct a lspec (for bus number 0).
Quoted 4 lines
The
of_irq_map_pcicode stops walking as soon as it finds a function in the tree that has a device node. This suggests that if you represent a bridge you must also include an interrupt-map. The problem here is that you have represented a bridge but not included a map.
I understood that it walks up the PCI hierarchy, and that's fine. As I've shown in my previous e-mail, the only problem is that this pdev->dev.of_node should be NULL for the PCIe endpoint device. If it were NULL, then everything would work correctly, as I could confirmed by the hack I did in of_irq_map_pci().
Quoted 14 lines
I can think of three solutions:
1. Something like this:
} else { /* We found a P2P bridge, check if it has a node */ ppnode = pci_device_to_OF_node(ppdev); + if (ppnode doesnt have an interrupt-map)//if (pdev->bus->number != 0) + ppnode = NULL; }2. Remove the bridges from the DT? Or remove the map from pcie-controller and add a map each to pcie at 0,1 and pcie at 1,1?
Quoted 8 lines
3. Change the mask of your map so that it doesn't care about bus numbers. I have a map that looks like this:
interrupt-map-mask = <0 0 0 7>; interrupt-map = <0 0 0 1 ... //anything coming in on INTA 0 0 0 2 ... //anything coming in on INTB 0 0 0 3 ... ... 0 0 0 4 ... ...
Unfortunately, I don't quite agree with any of your three solutions. I still do believe the root problem is that pdev->dev.of_node should be NULL for the PCIe endpoints, since those devices are not probed with the Device Tree.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Thomas Petazzoni,
Quoted 4 lines · Thomas Petazzoni
Unfortunately, I don't quite agree with any of your three solutions. I still do believe the root problem is that
pdev->dev.of_nodeshould be NULL for the PCIe endpoints, since those devices are not probed with the Device Tree.
Looking more at this, the pdev->dev.of_node is in fact NULL, but the code uses the of_node of the parent PCI device.
So in fact the problem is indeed that the subnodes pcie0,0 and pcie1,0 are seen as corresponding to the PCI-to-PCI bridges.
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Thomas Petazzoni
Dear Thomas Petazzoni,
Quoted 4 lines · Thomas Petazzoni
Unfortunately, I don't quite agree with any of your three solutions. I still do believe the root problem is that
pdev->dev.of_nodeshould be NULL for the PCIe endpoints, since those devices are not probed with the Device Tree.
Quoted 5 lines
Looking more at this, the
pdev->dev.of_nodeis in fact NULL, but the code uses theof_nodeof the parent PCI device.So in fact the problem is indeed that the subnodes pcie0,0 and pcie1,0 are seen as corresponding to the PCI-to-PCI bridges.
I would suggest changing the interrupt-mask to match any bus number. (Don't forget that the secondary bus number of each of your emulated bridges will vary depending on how many devices are detected underneath each root port, assuming you don't try and partition bus numbers or use domains between ports).
Andrew Murray
Quoted 10 lines
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 2 lines
So in fact the problem is indeed that the subnodes pcie0,0 and pcie1,0 are seen as corresponding to the PCI-to-PCI bridges.
I would suggest changing the interrupt-mask to match any bus number. (Don't forget that the secondary bus number of each of your emulated bridges will vary depending on how many devices are detected underneath each root port, assuming you don't try and partition bus numbers or use domains between ports).
I don't think this would work. Currently, the interrupt-map associates the interrupts with the PCI-to-PCI bridges, i.e devices 00:01, 00:02, 00:03, 00:04, 00:05, etc.
The real PCIe devices themselves are at 01:00, 02:00, 03:00, 04:00, 05:00. Each of them sit on a different bus, at devfn = 0.
So if I ignore the bus number, how could the PCI code find what is the matching interrupt?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 16 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 2 lines
So in fact the problem is indeed that the subnodes pcie0,0 and pcie1,0 are seen as corresponding to the PCI-to-PCI bridges.
I would suggest changing the interrupt-mask to match any bus number. (Don't forget that the secondary bus number of each of your emulated bridges will vary depending on how many devices are detected underneath each root port, assuming you don't try and partition bus numbers or use domains between ports).
I don't think this would work. Currently, the interrupt-map associates the interrupts with the PCI-to-PCI bridges, i.e devices 00:01, 00:02, 00:03, 00:04, 00:05, etc.
Quoted 5 lines
The real PCIe devices themselves are at 01:00, 02:00, 03:00, 04:00, 05:00. Each of them sit on a different bus, at devfn = 0.
So if I ignore the bus number, how could the PCI code find what is the matching interrupt?
Apologies if I've missed information about your hardware in the other discussion (I've tried to keep up) - does your hardware raise a single host interrupt for each pin regardless to which bridge they come in on - or do you separate A,B,C,D host interrupts for each bridge?
If you have only 4 interrupt sources for legacy interrupts then you shouldn't need to care which bus/device/function they were generated on (of_pci_map_irq takes care of this for you).
During enumeration an interrupt number should be assigned to each requesting device which reflects the pin (after swizzling) which will arrive at the host bridges. That interrupt number should be shared across all devices that requested the same pin (after swizzling) - i.e. shared interrupts. So all you need to do is map A,B,C,D interrupts with the interrupt they come into the CPU.
Andrew Murray
Quoted 10 lines
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 2 lines
So if I ignore the bus number, how could the PCI code find what is the matching interrupt?
Apologies if I've missed information about your hardware in the other discussion (I've tried to keep up) - does your hardware raise a single host interrupt for each pin regardless to which bridge they come in on - or do you separate A,B,C,D host interrupts for each bridge?
There are separate A,B,C,D interrupts for each PCIe interface, and each PCIe interface is represented by an emulated PCI-to-PCI bridge. See my interrupt-map:
interrupt-map = <0x0800 0 0 1 &mpic 58 0x1000 0 0 1 &mpic 59 0x1800 0 0 1 &mpic 60 0x2000 0 0 1 &mpic 61 0x2800 0 0 1 &mpic 62 0x3000 0 0 1 &mpic 63 0x3800 0 0 1 &mpic 64 0x4000 0 0 1 &mpic 65 0x4800 0 0 1 &mpic 99 0x5000 0 0 1 &mpic 103>;
Here I have 10 PCIe interfaces, and therefore 10 interrupts.
There is only one interrupt per PCIe interface, and for now, I don't distinguish A,B,C,D (I will do it later, it requires reading a register to know if the interrupt came from A, B, C or D, but that's a different problem).
Quoted 3 lines
If you have only 4 interrupt sources for legacy interrupts then you shouldn't need to care which bus/device/function they were generated on (
of_pci_map_irqtakes care of this for you).
No, I have interrupts per PCIe interface, so I really need to take care of the relation between the PCIe device and the PCIe interface it is connected to.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 27 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 2 lines
So if I ignore the bus number, how could the PCI code find what is the matching interrupt?
Apologies if I've missed information about your hardware in the other discussion (I've tried to keep up) - does your hardware raise a single host interrupt for each pin regardless to which bridge they come in on - or do you separate A,B,C,D host interrupts for each bridge?
There are separate A,B,C,D interrupts for each PCIe interface, and each PCIe interface is represented by an emulated PCI-to-PCI bridge. See my interrupt-map:
interrupt-map = <0x0800 0 0 1 &mpic 58 0x1000 0 0 1 &mpic 59 0x1800 0 0 1 &mpic 60 0x2000 0 0 1 &mpic 61 0x2800 0 0 1 &mpic 62 0x3000 0 0 1 &mpic 63 0x3800 0 0 1 &mpic 64 0x4000 0 0 1 &mpic 65 0x4800 0 0 1 &mpic 99 0x5000 0 0 1 &mpic 103>;
Quoted 6 lines
Here I have 10 PCIe interfaces, and therefore 10 interrupts.
There is only one interrupt per PCIe interface, and for now, I don't distinguish A,B,C,D (I will do it later, it requires reading a register to know if the interrupt came from A, B, C or D, but that's a different problem).
Right, someday you can have all 40 interrupts :)
This interrupt-map stuff is ugly, I looks like it was designed as a way for the firmware to communicate the per-device interrupt assignment to the OS - using it to describe a HW setup is a bit more tortured for sure..
Frankly, I think it should not be used in PCI-E drivers *at all* beacuse there is no need for an external interrupt routing description. [1]
PCI-E *only* has inband interrupt delivery, so the host driver has 100% of the information it needs to convert a INTx signal received on a link to a Linux interrupt number.
So in a PCI-E world translation should look like this:
- Start at the source device INTx
- Traverse up bridges performing INTx swizzling according to the PCI-E spec
- When the host bridge is reached call into the host driver and pass
- The INTx number
- The bus/device/function of last bridge traversed
- The host bridge driver returns a Linux interrupt number.
This *always* works for (compliant) PCI-E. An OF interrupt map is never, ever, needed. Providing a common mechanism can remove this complex OF stuff and ultimately help everyone :)
However, if interrupt-map is used, I strongly suspect it should be placed on each bridge node (as Andrew mentioned), and it should be very simple:
link@0 {
reg = <0x800 0 0 0 0>; // Bus 0, Dev 0x10, Fn 0
interrupt-mask = <0x0 0 0 7>;
interrupt-map = <0x0000 0 0 1 &mpic 58 // INTA
0x0000 0 0 2 &mpic 58 // INTB
0x0000 0 0 3 &mpic 58 // INTC
0x0000 0 0 4 &mpic 58>; // INTD
}Which hopefully says 'when you reach this bridge node, look only at the INTx pin number and convert it'.
I don't think you can make interrupt-map work were you have placed it..
Jason
1 - For PCI/PCI-X the physical INTx interrupt pins on any device,
anywhere in the system could by physically tied to any interrupt ingress in the system. Even if they are behind bridges. It is a total free for all.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 8 lines · Jason Gunthorpe
On Thu, Feb 07, 2013 at 06:37:43PM +0100, Thomas Petazzoni wrote:
Right, someday you can have all 40 interrupts :)
This interrupt-map stuff is ugly, I looks like it was designed as a way for the firmware to communicate the per-device interrupt assignment to the OS - using it to describe a HW setup is a bit more tortured for sure..
It is usually used for PCI (non-PCIe) systems, where you can have multiple slots on the same bus and each slot has its own four interrupt lines routed to an upstream interrupt controller, not necessarily the same one for each slot.
Quoted 8 lines
Frankly, I think it should not be used in PCI-E drivers *at all* beacuse there is no need for an external interrupt routing description. [1]
PCI-E *only* has inband interrupt delivery, so the host driver has 100% of the information it needs to convert a INTx signal received on a link to a Linux interrupt number.
Quoted 8 lines
So in a PCI-E world translation should look like this:
- Start at the source device INTx
- Traverse up bridges performing INTx swizzling according to the PCI-E spec
- When the host bridge is reached call into the host driver and pass
- The INTx number
- The bus/device/function of last bridge traversed
- The host bridge driver returns a Linux interrupt number.
It's the same as on PCI, except that there you stop when you reach the slot that is physically connected to the interrupt controller.
Quoted 8 lines
This *always* works for (compliant) PCI-E. An OF interrupt map is never, ever, needed. Providing a common mechanism can remove this complex OF stuff and ultimately help everyone :)
However, if interrupt-map is used, I strongly suspect it should be placed on each bridge node (as Andrew mentioned), and it should be very simple:
Quoted 8 lines
link at 0 {
reg = <0x8000 0 0 0>; // Bus 0, Dev0x10, Fn 0
interrupt-mask = <0x00 0 7>;
interrupt-map = <0x00000 0 1 &mpic 58 // INTA0x0000 0 0 2 &mpic 58 // INTB 0x0000 0 0 3 &mpic 58 // INTC 0x0000 0 0 4 &mpic 58>; // INTD }
The interrupt-map property only makes sense for the host bridge, not for bridges below it, which don't normally get represented in the device tree.
Quoted 5 lines
Jason
1 - For PCI/PCI-X the physical INTx interrupt pins on any device,
anywhere in the system could by physically tied to any interrupt ingress in the system. Even if they are behind bridges. It is a total free for all.
I would not be surprised to see system that have a PCIe to PCI bridge and slots behind that which are directly connected to an upstream interrupt controller. In fact, I believe I have worked on a system like that (IBM QS21 and QS22), and thankfully the interrupt-map property did let us model this correctly.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Arnd Bergmann
Quoted 8 lines
link at 0 {
reg = <0x8000 0 0 0>; // Bus 0, Dev0x10, Fn 0
interrupt-mask = <0x00 0 7>;
interrupt-map = <0x00000 0 1 &mpic 58 // INTA0x0000 0 0 2 &mpic 58 // INTB 0x0000 0 0 3 &mpic 58 // INTC 0x0000 0 0 4 &mpic 58>; // INTD }
Quoted 3 lines
The interrupt-map property only makes sense for the host bridge, not for bridges below it, which don't normally get represented in the device tree.
Linux scans up the PCI bus until it finds a PCI device with a matching OF node. It then constructs an interrupt map 'laddr' (ie the bus:dev.fn) for the child device of this OF node.
If you don't have any DT PCI nodes then this should always fold down to doing a lookup with bus=0, and device representing the 'slot' in legacy PCI.
However, as soon as you provide a node for a bridge in DT this halts the 'fold down' and goes to the interrupt-map with a device on the subordinate bus number of the bridge.
This makes *lots* of sense, if you have bridges providing bus slots then you include the bridge in DT to stop the 'fold down' at that known bridge, giving you a chance to see the interrupt wiring behind the bridge.
This matches the design of PCI - if you know how interrupts are hooked up then use that information, otherwise assume the INTx interrupts swizzle and search upward. This is how add-in cards with PCI bridges are supported.
This behavior seems complex, but sane to me. I wouldn't change it as Andrew suggested.
Thomas's problem is the presence of the static DT node for the root port bridge. Since the node is static you can't know what the runtime determined subordinate bus numbers will be, so there is no possible way to write an interrupt-map at the host bridge.
Putting the map in the bridge's DT node seems elegant and correct to me - the map is describing the actual hardware - the root port bridge is actually terminating INTx from downstream devices and converting them to CPU interrupts. (FWIW discrete HT to PCIe bridges do something similar)
If you imagine the case you alluded to, a PCI-E root port, connected to a PCI-E to PCI bridge, with 2 physical PCI bus slots. The interrupts for the 2 slots are routed to the CPU directly:
link at 0 {
reg = </* Bus 0, Dev 0x10, Fn 0 */>; // Root Port bridge
// Match on INTx (not used since the pci-bridge doesn't create inband INTx) interrupt-mask = <0x0 0 0 7>;
interrupt-map = <0x0000 0 0 1 &pic 0 // Inband INTA
0x0000 0 0 2 &pic 1 // Inband INTB
..
pci_bridge@0 {
reg = </* Bus 1, Dev 0x10, Fn 0 */>; // PCIe to PCI bridge
// Match on the device/slot and INTx pin
interrupt-mask = <0x7f 0 0 7>;
interrupt-map = <0x00xx 0 0 1 &pic 2 // Slot 0 physical INTA
0x00xx 0 0 1 &pic 3 // Slot 1 physical INTA
.. }
}
To me, this seems to be a much more accurate description of how the hardware is constructed then trying to cram all this information into the host bridge's interrupt map. It shows clearly where inband INTA messages arriving@the root port are directed as well as where the slot by slot out-of-band interrupt wires on the PCI bus are directed.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 22 lines · Jason Gunthorpe
Quoted 9 lines · Arnd Bergmann
Quoted 8 lines
link at 0 {
reg = <0x8000 0 0 0>; // Bus 0, Dev0x10, Fn 0
interrupt-mask = <0x00 0 7>;
interrupt-map = <0x00000 0 1 &mpic 58 // INTA0x0000 0 0 2 &mpic 58 // INTB 0x0000 0 0 3 &mpic 58 // INTC 0x0000 0 0 4 &mpic 58>; // INTD }Quoted 3 lines
The interrupt-map property only makes sense for the host bridge, not for bridges below it, which don't normally get represented in the device tree.
Linux scans up the PCI bus until it finds a PCI device with a matching OF node. It then constructs an interrupt map 'laddr' (ie the bus:dev.fn) for the child device of this OF node.
If you don't have any DT PCI nodes then this should always fold down to doing a lookup with bus=0, and device representing the 'slot' in legacy PCI.
Quoted 8 lines
However, as soon as you provide a node for a bridge in DT this halts the 'fold down' and goes to the interrupt-map with a device on the subordinate bus number of the bridge.
This makes *lots* of sense, if you have bridges providing bus slots then you include the bridge in DT to stop the 'fold down' at that known bridge, giving you a chance to see the interrupt wiring behind the bridge.
I would argue that it matters not so much what Linux does but what the standard says, but it seems they both agree with you in this
case: http://www.openfirmware.org/1275/practice/imap/imap0_9d.pdf
defines that "At any level in the interrupt tree, a mapping may need to take place between the child interrupt domain and the parent?s. This is represented by a new property called 'interrupt-map'".
Quoted 4 lines
This matches the design of PCI - if you know how interrupts are hooked up then use that information, otherwise assume the INTx interrupts swizzle and search upward. This is how add-in cards with PCI bridges are supported.
Note that the implicit swizzling was not part of the original PCI binding, which assumed that all devices were explicitly represented in the device tree, and we don't normally do that any more because PCI can be probed easily, and we cannot assume that all PCI BARs have been correctly assigned by the firmware before the OS is booting. Having the interrupt-map at PCI host controller node is convenient because it lets us define unit interrupt specifiers for devices that are not represented in the device tree themselves.
I think the key question here is whether there is just one interrupt domain across all bridges because the hardware requires the unit address to be unique, or whether each PCIe port has its own unit address space, and thereby interrupt domain that requires its own interrupt-map.
If there is just one domain, we have the choice whether to have one interrupt-map for the entire domain, or to have one interrupt map per PCIe port for the devices under that port. I would consider it more logical to have a single interrupt-map for the interrupt domain, because that is essentially what lets us describe the interrupt daomain as a whole.
Of course, if each port has its own domain, we have to have a separate interrupt map for each one.
Quoted 4 lines
Thomas's problem is the presence of the static DT node for the root port bridge. Since the node is static you can't know what the runtime determined subordinate bus numbers will be, so there is no possible way to write an interrupt-map at the host bridge.
Right, that is a problem if there are additional bridges. I guess we could represent all devices on bus 0 easily because their address would be fixed, but can't uniquely identify anything below them.
Quoted 11 lines
If you imagine the case you alluded to, a PCI-E root port, connected to a PCI-E to PCI bridge, with 2 physical PCI bus slots. The interrupts for the 2 slots are routed to the CPU directly:
link at 0 {
reg = </* Bus 0, Dev0x10, Fn 0 */>; // Root Port bridge// Match on INTx (not used since the pci-bridge doesn't create inband INTx) interrupt-mask = <
0x00 0 7>;
interrupt-map = <0x00000 0 1 &pic 0 // Inband INTA0x0000 0 0 2 &pic 1 // Inband INTB
What are these two interrupts in the example then?
Quoted 2 lines
pci_bridge at 0 {
reg = </* Bus 1, Dev 0x10, Fn 0 */>; // PCIe to PCI bridgeThe device would be "pci at 10", right?
Quoted 7 lines
// Match on the device/slot and INTx pin interrupt-mask = <0x7f 0 0 7>; interrupt-map = <0x00xx 0 0 1 &pic 2 // Slot 0 physical INTA 0x00xx 0 0 1 &pic 3 // Slot 1 physical INTA ..}
}
You are accidentally matching the on the register number, not the device number here, right? The interrupt-map-mask should be <0xf800 0 0 7> to match the device.
Quoted 5 lines
To me, this seems to be a much more accurate description of how the hardware is constructed then trying to cram all this information into the host bridge's interrupt map. It shows clearly where inband INTA messages arriving at the root port are directed as well as where the slot by slot out-of-band interrupt wires on the PCI bus are directed.
Yes, I guess you're right.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 11 lines · Arnd Bergmann
Quoted 4 lines
This makes *lots* of sense, if you have bridges providing bus slots then you include the bridge in DT to stop the 'fold down' at that known bridge, giving you a chance to see the interrupt wiring behind the bridge.
I would argue that it matters not so much what Linux does but what the standard says, but it seems they both agree with you in this
case: http://www.openfirmware.org/1275/practice/imap/imap0_9d.pdf
defines that "At any level in the interrupt tree, a mapping may need to take place between the child interrupt domain and the parent???s. This is represented by a new property called 'interrupt-map'".
Right, the standard (and Linux) allows the property in any node.
Quoted 8 lines
Quoted 4 lines
This matches the design of PCI - if you know how interrupts are hooked up then use that information, otherwise assume the INTx interrupts swizzle and search upward. This is how add-in cards with PCI bridges are supported.
Note that the implicit swizzling was not part of the original PCI binding, which assumed that all devices were explicitly represented in the device tree, and we don't normally do that any more because
Yes, if the tree includes all PCI devices then you don't need interrupt-map, just place an interrupt property directly on the end nodes.
However the implicit swizzling and 'fold down' is pretty much essential to support the PCI standards for hot plug behind bridges.
Quoted 6 lines
PCI can be probed easily, and we cannot assume that all PCI BARs have been correctly assigned by the firmware before the OS is booting. Having the interrupt-map at PCI host controller node is convenient because it lets us define unit interrupt specifiers for devices that are not represented in the device tree themselves.
Right, interrupt-map is actually only needed when the DT is incomplete, or to support hot pluggable ports.
Quoted 5 lines
I think the key question here is whether there is just one interrupt domain across all bridges because the hardware requires the unit address to be unique, or whether each PCIe port has its own unit address space, and thereby interrupt domain that requires its own interrupt-map.
IMHO, the interrupt domains should describe the underlying hardware, and in many cases the HW is designed so that every PCI bus bridge has it's own downstream interrupt layout - and thus is an interrupt domain.
Quoted 7 lines
Quoted 7 lines
If you imagine the case you alluded to, a PCI-E root port, connected to a PCI-E to PCI bridge, with 2 physical PCI bus slots. The interrupts for the 2 slots are routed to the CPU directly:
link at 0 {
reg = </* Bus 0, Dev0x10, Fn 0 */>; // Root Port bridge
Quoted 6 lines
Quoted 4 lines
// Match on INTx (not used since the pci-bridge doesn't create inband INTx) interrupt-mask = <
0x00 0 7>;
interrupt-map = <0x00000 0 1 &pic 0 // Inband INTA0x0000 0 0 2 &pic 1 // Inband INTBWhat are these two interrupts in the example then?
This shows that the HW block 'link at 0' - which is a PCI Express root port bridge - accepts inband INTx messages and converts them to CPU interrupts pic 0/1/...
Since this is a general function, and fully self contained, it can be placed in the general SOC's dtsi.
However, the board has a hard-wired PCIe to PCI bridge with PCI slots, and never generates inband INTx. We can then describe that chip via the following stanza in the board specific dts:
Quoted 4 lines
Quoted 2 lines
pci_bridge at 0 { reg = </* Bus 1, Dev 0x10, Fn 0 */>; // PCIe to PCI bridgeThe device would be "pci at 10", right?
Probably best to use the hex version of the regs value /* Bus 1, Dev 0x10, Fn 0 */, but nothing inspects that, right?
Quoted 11 lines
Quoted 7 lines
// Match on the device/slot and INTx pin interrupt-mask = <0x7f 0 0 7>; interrupt-map = <0x00xx 0 0 1 &pic 2 // Slot 0 physical INTA 0x00xx 0 0 1 &pic 3 // Slot 1 physical INTA ..}
}You are accidentally matching the on the register number, not the device number here, right? The interrupt-map-mask should be <
0xf8000 0 7> to match the device.
Right, only match the device, ignore the bus.
There is also another variant, if the PCIe to PCI bridge provides its own interrupt pins and converts those to inband PCIe INTx messages, then the PCB can wire up the PCI bus slots to the bridge's INTx pins according to some pattern and describe that pattern in DT:
pci_bridge at 0 {
reg = </* Bus 1, Dev 0x10, Fn 0 */>; // PCIe to PCI bridge
interrupt-mask = <0xf800 0 0 7>;
interrupt-map = <0x00xx 0 0 1 &pci_bridge0 0 0 0 1 // Slot 0 physical INTA to inband INTA
0x00xx 0 0 1 &pci_bridge0 0 0 0 2 // Slot 1 physical INTA to inband INTB ... }
(minus errors, haven't tried this one, but the standard says it should be OK)
Which would be processed as:
- pci_bridge at 0 converts out of brand interrupts into in-band interrupts according its interrupt-map, and then sends those upstream.
- link@0 converts in band interrupts into CPU interrupts according to its interrupt map.
In my experience the above is a common case.
Boot firmware could fold all this down to a single interrupt map, and hide the programming of the IOAPIC/etc from the OS, but the HW is still undertaking these transformations..
Anyhow, it sounds like Thomas has had success using this approach, so it works.
Cheers,
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 34 lines · Thomas Petazzoni
Dear Andrew Murray,
Quoted 7 lines · Andrew Murray
Quoted 2 lines
So if I ignore the bus number, how could the PCI code find what is the matching interrupt?
Apologies if I've missed information about your hardware in the other discussion (I've tried to keep up) - does your hardware raise a single host interrupt for each pin regardless to which bridge they come in on - or do you separate A,B,C,D host interrupts for each bridge?
There are separate A,B,C,D interrupts for each PCIe interface, and each PCIe interface is represented by an emulated PCI-to-PCI bridge. See my interrupt-map:
interrupt-map = <0x0800 0 0 1 &mpic 58 0x1000 0 0 1 &mpic 59 0x1800 0 0 1 &mpic 60 0x2000 0 0 1 &mpic 61 0x2800 0 0 1 &mpic 62 0x3000 0 0 1 &mpic 63 0x3800 0 0 1 &mpic 64 0x4000 0 0 1 &mpic 65 0x4800 0 0 1 &mpic 99 0x5000 0 0 1 &mpic 103>;Here I have 10 PCIe interfaces, and therefore 10 interrupts.
There is only one interrupt per PCIe interface, and for now, I don't distinguish A,B,C,D (I will do it later, it requires reading a register to know if the interrupt came from A, B, C or D, but that's a different problem).
Quoted 7 lines
Quoted 3 lines
If you have only 4 interrupt sources for legacy interrupts then you shouldn't need to care which bus/device/function they were generated on (
of_pci_map_irqtakes care of this for you).No, I have interrupts per PCIe interface, so I really need to take care of the relation between the PCIe device and the PCIe interface it is connected to.
In that case, I think you can create a mask that only checks for the device number and INT pin (i.e. ignore bus and function). Looking at your mask - it already does this...
interrupt-map = <0x0800 0 0 1 &mpic 58
0x1000 0 0 1 &mpic 59
0x1800 0 0 1 &mpic 60
0x2000 0 0 1 &mpic 61
0x2800 0 0 1 &mpic 62
0x3000 0 0 1 &mpic 63
0x3800 0 0 1 &mpic 64
0x4000 0 0 1 &mpic 65
0x4800 0 0 1 &mpic 99
0x5000 0 0 1 &mpic 103>;I'm not sure if the device pin part of the map is correct (I always forget how this works) - but I know this would definately work:
interrupt-map-mask = <0xf800 0 0 7>
interrupt-map = <0x0800 0 0 1 &mpic 58
0x0800 0 0 2 &mpic 58
0x0800 0 0 3 &mpic 58
0x0800 0 0 4 &mpic 58
0x1000 0 0 1 &mpic 59
0x1000 0 0 2 &mpic 59
0x1000 0 0 3 &mpic 59
0x1000 0 0 4 &mpic 59
....In any case, I've realized that my original suggestion of changing the map won't quite do (apologies). This is because the OF code won't even look at this map as it stops@the emulated bridge below. In addition to this type of mapping - you'll also need to investigate my solution 1 and 2.
Andrew Murray
Quoted 11 lines
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 13 lines · Andrew Murray
I'm not sure if the device pin part of the map is correct (I always forget how this works) - but I know this would definately work:
interrupt-map-mask = <0xf800 0 0 7> interrupt-map = <0x0800 0 0 1 &mpic 58 0x0800 0 0 2 &mpic 58 0x0800 0 0 3 &mpic 58 0x0800 0 0 4 &mpic 58 0x1000 0 0 1 &mpic 59 0x1000 0 0 2 &mpic 59 0x1000 0 0 3 &mpic 59 0x1000 0 0 4 &mpic 59 ....
If all the pins are routed to the same interrupt, you can leave the pin out of the map-mask and save a bunch of lines in the interrupt-map, at least by the spec. I have not looked at the source code to see if that is how Linux implements the lookup, but that can be fixed if necessary.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 5 lines · Thomas Petazzoni
drivers/pci/host/pci-mvebu.c new file +5
-0+static resource_size_t mvebu_pcie_align_resource (struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align)
drivers/pci/host/pci-mvebu.c line 335
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+{ +if (!(res->flags &IORESOURCE_IO )) +return start; + +/* +* The I/O regions must be 64K aligned, because the +* granularity of PCIe I/O address decoding windows is 64 K +*/ +return round_up (start,SZ_64K ); +}
You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
I've just been digging through the PCI code and have come across a function - pcibios_window_alignment() - which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.
I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Russell,
Quoted 2 lines · Russell King - ARM Linux
You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
Hum, yes, correct.
Quoted 8 lines
I've just been digging through the PCI code and have come across a function -
pcibios_window_alignment()- which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
Seems interesting indeed, I'll look into this idea! Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Russell King - ARM LinuxQuoted 20 lines · Russell King - ARM Linux
Quoted 5 lines · Thomas Petazzoni
drivers/pci/host/pci-mvebu.c new file +5
-0+static resource_size_t mvebu_pcie_align_resource (struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align)drivers/pci/host/pci-mvebu.c line 335
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+{ +if (!(res->flags &IORESOURCE_IO )) +return start; + +/* +* The I/O regions must be 64K aligned, because the +* granularity of PCIe I/O address decoding windows is 64 K +*/ +return round_up (start,SZ_64K ); +}You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
Quoted 8 lines
I've just been digging through the PCI code and have come across a function -
pcibios_window_alignment()- which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
That approach isn't going to work very well with multi-platform, though, since the function can only be overridden on a per-architecture basis.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130130/74cffa44/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Thierry Reding,
Quoted 3 lines · Thierry Reding
That approach isn't going to work very well with multi-platform, though, since the function can only be overridden on a per-architecture basis.
I can do like is done for pcibios_align_resource(): put a single implementation of the function in arch/arm/kernel/bios32.c, and make it call a hook registered in the hw_pci structure, or even directly use a numerical value in hw_pci, as Russell suggested earlier.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thierry RedingQuoted 26 lines · Thierry Reding
Quoted 25 lines · Russell King - ARM Linux
Quoted 5 lines · Thomas Petazzoni
drivers/pci/host/pci-mvebu.c new file +5
-0+static resource_size_t mvebu_pcie_align_resource (struct pci_dev *dev, +const struct resource *res, +resource_size_t start, +resource_size_t size, +resource_size_t align)drivers/pci/host/pci-mvebu.c line 335
Quoted 10 lines
drivers/pci/host/pci-mvebu.c new file +10
-0+{ +if (!(res->flags &IORESOURCE_IO )) +return start; + +/* +* The I/O regions must be 64K aligned, because the +* granularity of PCIe I/O address decoding windows is 64 K +*/ +return round_up (start,SZ_64K ); +}You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
I've just been digging through the PCI code and have come across a function -
pcibios_window_alignment()- which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.
Quoted 6 lines
Quoted 3 lines
I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
That approach isn't going to work very well with multi-platform, though, since the function can only be overridden on a per-architecture basis.
The same can be said of all the various other functions which the PCI stuff expects the arch to provide, yet we seem to cope just fine...
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 31 lines · Russell King - ARM Linux
Quoted 30 lines · Thierry Reding
Quoted 28 lines · Russell King - ARM Linux
Quoted 5 lines · Thomas Petazzoni
(quoted diff) +5
-0+static resource_size_t mvebu_pcie_align_resource(struct pci_dev *dev, + const struct resource *res, + resource_size_t start, + resource_size_t size, + resource_size_t align)Review
Quoted 10 lines
(quoted diff) +10
-0+{ + if (!(res->flags & IORESOURCE_IO)) + return start; + + /* + * The I/O regions must be 64K aligned, because the + * granularity of PCIe I/O address decoding windows is 64 K + */ + return round_up(start, SZ_64K); +}You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
I've just been digging through the PCI code and have come across a function -
pcibios_window_alignment()- which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
Quoted 5 lines
Quoted 2 lines
That approach isn't going to work very well with multi-platform, though, since the function can only be overridden on a per-architecture basis.
The same can be said of all the various other functions which the PCI stuff expects the arch to provide, yet we seem to cope just fine...
And this (untested) is how it's done:
arch/arm/include/asm/mach/pci.h | 1 + arch/arm/kernel/bios32.c | 8 ++++++++ 2 files changed, 9 insertions(+), 0 deletions(-)
arch/arm/include/asm/mach/pci.h +1 -0
@@ -29,6 +29,7 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*window_align)(struct pci_bus *,unsigned long ); };/*
arch/arm/kernel/bios32.c +8 -0
@@ -581,6 +581,14 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,return start; } +resource_size_t pcibios_window_alignment (struct pci_bus *bus, +unsigned long type) +{ +struct pci_sys_data *sys = bus->sysdata; + +return sys->window_alignment ? sys->window_alignment (bus, type) :1 ; +} +/** * pcibios_enable_device - Enable I/O and memory. * @dev: PCI device to be enabled
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Russell King - ARM Linux,
arch/arm/include/asm/mach/pci.h struct hw_pci { · line 33
Quoted 10 lines · Russell King - ARM Linux
arch/arm/include/asm/mach/pci.h +1
-0@@ -29,6 +29,7 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*window_align)(struct pci_bus *,unsigned long ); };
Didn't you say just yesterday that you would prefer a numerical value rather than a hook that could do random things? :-)
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 16 lines · Thomas Petazzoni
Dear Russell King - ARM Linux,
arch/arm/include/asm/mach/pci.h struct hw_pci { · line 33
Quoted 10 lines · Russell King - ARM Linux
arch/arm/include/asm/mach/pci.h +1
-0@@ -29,6 +29,7 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*window_align)(struct pci_bus *,unsigned long ); };Didn't you say just yesterday that you would prefer a numerical value rather than a hook that could do random things? :-)
Hrh, right. :)
And it also helps if I modify the right struct too! Try this instead. Overwrite sys->win_align_mem / sys->win_align_io in the setup function as required.
arch/arm/include/asm/mach/pci.h | 2 ++ arch/arm/kernel/bios32.c | 17 +++++++++++++++++ 2 files changed, 19 insertions(+), 0 deletions(-)
arch/arm/include/asm/mach/pci.h +2 -0
@@ -42,6 +42,8 @@ struct pci_sys_data {int busnr;/* primary bus number */ u64 mem_offset;/* bus->cpu memory mapping offset */ unsigned long io_offset;/* bus->cpu IO mapping offset */ +resource_size_t win_align_mem; +resource_size_t win_align_io;struct pci_bus *bus;/* PCI bus */ struct list_head resources;/* root bus resources (apertures) */ struct resource io_res;
arch/arm/kernel/bios32.c +17 -0
@@ -463,6 +463,8 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head) sys->swizzle = hw->swizzle; sys->map_irq = hw->map_irq;INIT_LIST_HEAD (&sys->resources); + sys->win_align_mem =1 ; + sys->win_align_io =1 ; ret = hw->setup (nr, sys);
@@ -581,6 +583,21 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,return start; } +resource_size_t pcibios_window_alignment (struct pci_bus *bus, +unsigned long type) +{ +struct pci_sys_data *sys = bus->sysdata; + +/* Ignore downstream buses */ +if (!bus->parent) { +if (type &IORESOURCE_MEM ) +return sys->win_align_mem; +if (type &IORESOURCE_IO ) +return sys->win_align_io; + } +return 1 ; +} +/** * pcibios_enable_device - Enable I/O and memory. * @dev: PCI device to be enabled
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Russell King - ARM Linux,
Quoted 5 lines · Russell King - ARM Linux
arch/arm/kernel/bios32.c +5
-0+resource_size_t pcibios_window_alignment (struct pci_bus *bus, +unsigned long type) +{ +struct pci_sys_data *sys = bus->sysdata; +
arch/arm/kernel/bios32.c resource_size_t pcibios_align_resource(void *data, const struct resource *res, · line 600
Quoted 10 lines
arch/arm/kernel/bios32.c +10
-0+/* Ignore downstream buses */ +if (!bus->parent) { +if (type &IORESOURCE_MEM ) +return sys->win_align_mem; +if (type &IORESOURCE_IO ) +return sys->win_align_io; + } +return 1 ; +} +
Unfortunately, this doesn't work as is for me: the if (!bus->parent) prevents the thing from being effective. Here my lspci output:
# /usr/sbin/lspci 00:00.0 Host bridge: Marvell Technology Group Ltd. Device 102d 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 03:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02) 05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
And the function pcibios_window_alignment() only gets called for bus 1, 2, 3, 4, 5, 6 and never for bus 0. And therefore, the !bus->parent test is always false. So, if I keep your implementation, I have the following addresses assigned to my bridges:
pci 0000:00:03.0: PCI bridge to [bus 03]
pci 0000:00:03.0: bridge window [io 0xc0000000-0xc0000fff] pci 0000:00:03.0: bridge window [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: bridge window [mem 0xc1100000-0xc11fffff pref] pci 0000:00:05.0: PCI bridge to [bus 05] pci 0000:00:05.0: bridge window [io 0xc0001000-0xc0001fff] pci 0000:00:05.0: bridge window [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: bridge window [mem 0xc1300000-0xc13fffff pref]
Notice how the io window of the second bridge starts 4K after the io window of the first bridge. Which cannot work on Marvell SoC, due to the 64KB alignment.
If, however, I remove the !bus->parent test, the I/O addresses correctly take into account the 64K requirement:
pci 0000:00:03.0: PCI bridge to [bus 03]
pci 0000:00:03.0: bridge window [io 0xc0000000-0xc000ffff] pci 0000:00:03.0: bridge window [mem 0xc1000000-0xc10fffff] pci 0000:00:03.0: bridge window [mem 0xc1100000-0xc11fffff pref] pci 0000:00:05.0: PCI bridge to [bus 05] pci 0000:00:05.0: bridge window [io 0xc0010000-0xc001ffff] pci 0000:00:05.0: bridge window [mem 0xc1200000-0xc12fffff] pci 0000:00:05.0: bridge window [mem 0xc1300000-0xc13fffff pref]
Any idea?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 34 lines · Thomas Petazzoni
Dear Russell King - ARM Linux,
Quoted 5 lines · Russell King - ARM Linux
arch/arm/kernel/bios32.c +5
-0+resource_size_t pcibios_window_alignment (struct pci_bus *bus, +unsigned long type) +{ +struct pci_sys_data *sys = bus->sysdata; +arch/arm/kernel/bios32.c resource_size_t pcibios_align_resource(void *data, const struct resource *res, · line 600
Quoted 10 lines
arch/arm/kernel/bios32.c +10
-0+/* Ignore downstream buses */ +if (!bus->parent) { +if (type &IORESOURCE_MEM ) +return sys->win_align_mem; +if (type &IORESOURCE_IO ) +return sys->win_align_io; + } +return 1 ; +} +Unfortunately, this doesn't work as is for me: the if (!bus->parent) prevents the thing from being effective. Here my lspci output:
# /usr/sbin/lspci 00:00.0 Host bridge: Marvell Technology Group Ltd. Device 102d 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 03:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02) 05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Quoted 2 lines
And the function
pcibios_window_alignment()only gets called for bus 1, 2, 3, 4, 5, 6 and never for bus 0.
That's the exact reverse of what I'd expect: the child buses should have a non-NULL parent pointer. Hmm. Try changing that for !bus->self - that should make it effective only on the host bridge.
But... hang on...
/*
* Returns true if the pci bus is root (behind host-pci bridge),
* false otherwise
*/
static inline bool pci_is_root_bus(struct pci_bus *pbus)
{
return !(pbus->parent);
}So the original test _is_ correct, and should only be triggered for the _root_ bus, that being bus 0 in the above case.
But... wait a moment, what are you saying? Which bridges need this fixup? The Marvell PCI-to-PCI bridges or the host bridge?
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Russell King - ARM Linux,
Quoted 25 lines · Russell King - ARM Linux
Quoted 25 lines
arch/arm/kernel/bios32.c resource_size_t pcibios_align_resource(void *data, const struct resource *res, · line 600
Quoted 10 lines
arch/arm/kernel/bios32.c +10
-0+/* Ignore downstream buses */ +if (!bus->parent) { +if (type &IORESOURCE_MEM ) +return sys->win_align_mem; +if (type &IORESOURCE_IO ) +return sys->win_align_io; + } +return 1 ; +} +Unfortunately, this doesn't work as is for me: the if (!bus->parent) prevents the thing from being effective. Here my lspci output:
# /usr/sbin/lspci 00:00.0 Host bridge: Marvell Technology Group Ltd. Device 102d 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:04.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 00:06.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 03:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02) 05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Quoted 5 lines
Quoted 2 lines
And the function
pcibios_window_alignment()only gets called for bus 1, 2, 3, 4, 5, 6 and never for bus 0.That's the exact reverse of what I'd expect: the child buses should have a non-NULL parent pointer.
Indeed. But this function never gets called with bus->number == 0, only with bus->number = 1, 2, 3, 4, 5, 6. So those are child busses, and therefore they have a parent.
If I had a debug message in this pcibios_window_alignment() function (which gets shown unconditionally, i.e the debug message is outside the if condition we are discussing), then I get:
pcibios_window_alignment: called for bus ef371c00 (sysdata=ef2f6bc0), number 1pcibios_window_alignment: called for bus ef371c00 (sysdata=ef2f6bc0), number 1pcibios_window_alignment: called for bus ef371c00 (sysdata=ef2f6bc0), number 1pcibios_window_alignment: called for bus ef371a00 (sysdata=ef2f6bc0), number 2pcibios_window_alignment: called for bus ef371a00 (sysdata=ef2f6bc0), number 2pcibios_window_alignment: called for bus ef371a00 (sysdata=ef2f6bc0), number 2pcibios_window_alignment: called for bus ef371800 (sysdata=ef2f6bc0), number 3pcibios_window_alignment: called for bus ef371800 (sysdata=ef2f6bc0), number 3pcibios_window_alignment: called for bus ef371800 (sysdata=ef2f6bc0), number 3pcibios_window_alignment: called for bus ef371600 (sysdata=ef2f6bc0), number 4pcibios_window_alignment: called for bus ef371600 (sysdata=ef2f6bc0), number 4pcibios_window_alignment: called for bus ef371600 (sysdata=ef2f6bc0), number 4pcibios_window_alignment: called for bus ef371400 (sysdata=ef2f6bc0), number 5pcibios_window_alignment: called for bus ef371400 (sysdata=ef2f6bc0), number 5pcibios_window_alignment: called for bus ef371400 (sysdata=ef2f6bc0), number 5pcibios_window_alignment: called for bus ef371200 (sysdata=ef2f6bc0), number 6pcibios_window_alignment: called for bus ef371200 (sysdata=ef2f6bc0), number 6pcibios_window_alignment: called for bus ef371200 (sysdata=ef2f6bc0), number 6
See, never called bus bus number 0.
Quoted 10 lines
Hmm. Try changing that for !bus->self -
that should make it effective only on the host bridge.But... hang on...
/*
* Returns true if the pci bus is root (behind host-pci bridge),
* false otherwise
*/
static inline boolpci_is_root_bus(struct pci_bus *pbus)
Quoted 6 lines
{ return !(pbus->parent); }So the original test _is_ correct, and should only be triggered for the _root_ bus, that being bus 0 in the above case.
Except that this pcibios_window_alignement() function is apparently never called for the root bus.
Quoted 3 lines
But... wait a moment, what are you saying? Which bridges need this fixup? The Marvell PCI-to-PCI bridges or the host bridge?
I am talking about the PCI-to-PCI bridges. I want the I/O windows assigned to each PCI-to-PCI bridge to be 64K aligned. The PCI-to-PCI bridges are devices that sit on bus 0, each giving access to the child buses 1, 2, 3, 4, 5, 6.
I have the impression that the pcibios_window_alignment() function is called on the *child* bus to know the requested alignments for the bridge that sits on the parent bus and gives access to this child bus.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 3 lines · Thomas Petazzoni
Indeed. But this function never gets called with
bus->number== 0, only withbus->number= 1, 2, 3, 4, 5, 6. So those are child busses, and therefore they have a parent.
Having thought about it, yes, that's what I expect, because root bus resources are never resized. Root bus resources are normally setup before probing to indicate what size they _can_ be and define what space is available to the downstream devices.
Child busses (behind a PCI-2-PCI bridge) are a different matter - these will be adjusted according to their on-bus devices and the windows for them sized and allocated appropriately _within_ the confines of the root bus resource.
Quoted 4 lines
I am talking about the PCI-to-PCI bridges. I want the I/O windows assigned to each PCI-to-PCI bridge to be 64K aligned. The PCI-to-PCI bridges are devices that sit on bus 0, each giving access to the child buses 1, 2, 3, 4, 5, 6.
Right, so you've just confirmed that this _is_ the right hook and it _is_ being called at the right time.
However, I had interpreted your requirement as the _host_ bridge only (insufficient information in your previous emails, or I missed it).
If that's what your bridge requires, then we need to detect it via its vendor and device IDs and only apply this fixup to those bridges which require a 64K alignment.
So, the IDs are vendor:device = 0x11ab:0x1092 ? And let me get this straight, it _is_ a specific requirement for this particular bridge P2P bridge?
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Russell King - ARM Linux,
Quoted 10 lines · Russell King - ARM Linux
Quoted 3 lines · Thomas Petazzoni
Indeed. But this function never gets called with
bus->number== 0, only withbus->number= 1, 2, 3, 4, 5, 6. So those are child busses, and therefore they have a parent.Having thought about it, yes, that's what I expect, because root bus resources are never resized. Root bus resources are normally setup before probing to indicate what size they _can_ be and define what space is available to the downstream devices.
Quoted 4 lines
Child busses (behind a PCI-2-PCI bridge) are a different matter - these will be adjusted according to their on-bus devices and the windows for them sized and allocated appropriately _within_ the confines of the root bus resource.
Ok.
Quoted 7 lines
Quoted 4 lines
I am talking about the PCI-to-PCI bridges. I want the I/O windows assigned to each PCI-to-PCI bridge to be 64K aligned. The PCI-to-PCI bridges are devices that sit on bus 0, each giving access to the child buses 1, 2, 3, 4, 5, 6.
Right, so you've just confirmed that this _is_ the right hook and it _is_ being called at the right time.
Ok.
Quoted 2 lines
However, I had interpreted your requirement as the _host_ bridge only (insufficient information in your previous emails, or I missed it).
Might be insufficient information, or wrong terminology on my side.
Quoted 7 lines
If that's what your bridge requires, then we need to detect it via its vendor and device IDs and only apply this fixup to those bridges which require a 64K alignment.
So, the IDs are vendor:device = 0x11ab:0x1092 ? And let me get this straight, it _is_ a specific requirement for this particular bridge P2P bridge?
The vendor/device ID will change. This P2P bridge is emulated. However, I'm not sure why you need to know the vendor:device ID to make the fixup specific.
The fixup is already specific to those bridges, since I am just setting pci_sys_data->win_align_io to 64K for the particular buses that are downstream the problematic bridges. So it doesn't affect any other bus on the system, and therefore I don't think this fixup needs to be made specific to a given vendor:device, no?
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 15 lines · Thomas Petazzoni
Dear Russell King - ARM Linux,
Quoted 7 lines · Russell King - ARM Linux
If that's what your bridge requires, then we need to detect it via its vendor and device IDs and only apply this fixup to those bridges which require a 64K alignment.
So, the IDs are vendor:device = 0x11ab:0x1092 ? And let me get this straight, it _is_ a specific requirement for this particular bridge P2P bridge?
The vendor/device ID will change. This P2P bridge is emulated. However, I'm not sure why you need to know the vendor:device ID to make the fixup specific.
Quoted 5 lines
The fixup is already specific to those bridges, since I am just setting
pci_sys_data->win_align_ioto 64K for the particular buses that are downstream the problematic bridges. So it doesn't affect any other bus on the system, and therefore I don't think this fixup needs to be made specific to a given vendor:device, no?
The pci_sys_data is not specific to one bus. It's specific from the root bus downwards, and is shared by all child busses.
The problem is if you have some card or a conventional P2P bridge which has 4K windows. If you merely set the alignment to 64K for all bridges, then all bridges get this treatment whether or not they need it. That's what I'm trying to avoid.
Take, for instance, a cardbus bridge (remember, there are PCI cards which can be plugged in to give you a cardbus slot.) I have a device here which can be plugged into a cardbus slot which has not just one P2P bridge but two, and a bunch of downsteam devices, including VGA, ethernet, USB, PS/2 etc. (Okay, Linux doesn't support this hardware because of crappy X86 stuff, despite the fact Windows cope with it just fine.)
There have been cards in the past which have had P2P bridges on them as well.
So, simply believing that the only P2P bridges in the system will be those on the physical board is a mistake.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Russell King - ARM Linux,
Quoted 8 lines · Russell King - ARM Linux
Quoted 5 lines
The fixup is already specific to those bridges, since I am just setting
pci_sys_data->win_align_ioto 64K for the particular buses that are downstream the problematic bridges. So it doesn't affect any other bus on the system, and therefore I don't think this fixup needs to be made specific to a given vendor:device, no?The
pci_sys_datais not specific to one bus. It's specific from the root bus downwards, and is shared by all child busses.
Ah, ok, that's the part I was missing.
Quoted 12 lines
The problem is if you have some card or a conventional P2P bridge which has 4K windows. If you merely set the alignment to 64K for all bridges, then all bridges get this treatment whether or not they need it. That's what I'm trying to avoid.
Take, for instance, a cardbus bridge (remember, there are PCI cards which can be plugged in to give you a cardbus slot.) I have a device here which can be plugged into a cardbus slot which has not just one P2P bridge but two, and a bunch of downsteam devices, including VGA, ethernet, USB, PS/2 etc. (Okay, Linux doesn't support this hardware because of crappy X86 stuff, despite the fact Windows cope with it just fine.)
Quoted 5 lines
There have been cards in the past which have had P2P bridges on them as well.
So, simply believing that the only P2P bridges in the system will be those on the physical board is a mistake.
Yes, indeed, I understand this. I just thought this pci_sys_data structure was per-bus. Of course, if it's shared by all buses on the system, we need a way to apply this fixup only to the Marvell bridges.
Should I just hard-code this special fixup in pcibios_window_alignment() with a check on VID/PID ?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 33 lines · Thomas Petazzoni
Dear Russell King - ARM Linux,
Quoted 8 lines · Russell King - ARM Linux
Quoted 5 lines
The fixup is already specific to those bridges, since I am just setting
pci_sys_data->win_align_ioto 64K for the particular buses that are downstream the problematic bridges. So it doesn't affect any other bus on the system, and therefore I don't think this fixup needs to be made specific to a given vendor:device, no?The
pci_sys_datais not specific to one bus. It's specific from the root bus downwards, and is shared by all child busses.Ah, ok, that's the part I was missing.
Quoted 12 lines
The problem is if you have some card or a conventional P2P bridge which has 4K windows. If you merely set the alignment to 64K for all bridges, then all bridges get this treatment whether or not they need it. That's what I'm trying to avoid.
Take, for instance, a cardbus bridge (remember, there are PCI cards which can be plugged in to give you a cardbus slot.) I have a device here which can be plugged into a cardbus slot which has not just one P2P bridge but two, and a bunch of downsteam devices, including VGA, ethernet, USB, PS/2 etc. (Okay, Linux doesn't support this hardware because of crappy X86 stuff, despite the fact Windows cope with it just fine.)
Quoted 5 lines
There have been cards in the past which have had P2P bridges on them as well.
So, simply believing that the only P2P bridges in the system will be those on the physical board is a mistake.
Quoted 6 lines
Yes, indeed, I understand this. I just thought this
pci_sys_datastructure was per-bus. Of course, if it's shared by all buses on the system, we need a way to apply this fixup only to the Marvell bridges.Should I just hard-code this special fixup in
pcibios_window_alignment()with a check on VID/PID ?
Yes please.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Russell King - ARM LinuxQuoted 15 lines · On Thursday 31 January 2013 15:36:25 Russell King - ARM Linux wrote:
The
pci_sys_datais not specific to one bus. It's specific from the root bus downwards, and is shared by all child busses.The problem is if you have some card or a conventional P2P bridge which has 4K windows. If you merely set the alignment to 64K for all bridges, then all bridges get this treatment whether or not they need it. That's what I'm trying to avoid.
Take, for instance, a cardbus bridge (remember, there are PCI cards which can be plugged in to give you a cardbus slot.) I have a device here which can be plugged into a cardbus slot which has not just one P2P bridge but two, and a bunch of downsteam devices, including VGA, ethernet, USB, PS/2 etc. (Okay, Linux doesn't support this hardware because of crappy X86 stuff, despite the fact Windows cope with it just fine.)
Quoted 5 lines
There have been cards in the past which have had P2P bridges on them as well.
So, simply believing that the only P2P bridges in the system will be those on the physical board is a mistake.
I was going to write something similar. Actually I think it's worse because the case of an extra P2P bridge is quite likely for devices that actually use I/O space, given that the use of I/O space is deprecated on PCIe.
This also means that a lot of devices using I/O space are legacy crap and have random bugs regarding PCI standard compliance. I would not expect those devices in general to do the right thing when I/O ports beyond 65535 are used, although a lot of them would work.
For all I could tell, the safest solution with the I/O space would be to pretend we had a shared 64K I/O space for all of the PCIe ports on Armada XP, and map a separate 64K window for each port using a different io_offset for each one.
This way, you can have a device on the second PCIe port use e.g. I/O port number 0x3f8 for a legacy UART on the bus, which gets translated into the Linux-visible port number 0x103f8.
The currently used method to have io_offset=0 for all PCIe ports and use separate I/O port ranges of 64K for each PCIe port probably still works for most devices, except those where we hardcode a port number in the Linux device driver, or where the high address bits don't get decoded properly.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 8 lines · Arnd Bergmann
For all I could tell, the safest solution with the I/O space would be to pretend we had a shared 64K I/O space for all of the PCIe ports on Armada XP, and map a separate 64K window for each port using a different
io_offsetfor each one.
This way, you can have a device on the second PCIe port use e.g. I/O port number0x3f8for a legacy UART on the bus, which gets translated into the Linux-visible port number0x103f8.
Quoted 5 lines
The currently used method to have io_offset=0 for all PCIe ports and use separate I/O port ranges of 64K for each PCIe port probably still works for most devices, except those where we hardcode a port number in the Linux device driver, or where the high address bits don't get decoded properly.
Thinking about this some more, which of these methods to choose is going to be dictated by what the Marvell HW does.
Since the IO space in the TLP is a full 32 bits, it matters what 32 bit value the HW PCI core places in the IO Rd/Wr transaction. This value *must* match the value given to the Linux PCI core for resource allocation, because it must be correctly programmed by Linux into the downstream BARs/bridge windows.
So there are probably two choices for what the HW does, given a MBUS window of 0xDEAD0000 -> 0xDEADFFFF set for IO, a read from physical address 0xDEAD0000 produces a IO Rd TLP with either '0x00000000' or '0xDEAD0000' in the address field.
If it is 0xDEAD0000, then Thomas has to keep what he has now, you can't mess with this address. Verify that the full 32 bit address exactly matching the MBUS window address is written to the PCI-PCI bridge IO base/limit registers.
If it is 0x00000000 then the mmap scheme I outlined before must be used, and verify that only 0->0xFFFF is written to the PCI-PCI bridge IO base/limit registers..
My guess is the Marvell PCI-E copies whatever address is it given into the IO TLP, so it would be the 0xDEAD0000 behaviour, however I bet you can use the MBUS window target address remapping feature to get the 0x00000000 behaviour as well (though there are a limited number of remappable MBUS windows, so that is probably not a good idea)
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Jason Gunthorpe
Thinking about this some more, which of these methods to choose is going to be dictated by what the Marvell HW does.
Since the IO space in the TLP is a full 32 bits, it matters what 32 bit value the HW PCI core places in the IO Rd/Wr transaction. This value must match the value given to the Linux PCI core for resource allocation, because it must be correctly programmed by Linux into the downstream BARs/bridge windows.
So there are probably two choices for what the HW does, given a MBUS window of
0xDEAD0000->0xDEADFFFFset for IO, a read from physical address0xDEAD0000produces a IO Rd TLP with either '0x00000000' or '0xDEAD0000' in the address field.
Quoted 4 lines
If it is
0xDEAD0000, then Thomas has to keep what he has now, you can't mess with this address. Verify that the full 32 bit address exactly matching the MBUS window address is written to the PCI-PCI bridge IO base/limit registers.
If you do this, you break all sorts of expectations in the kernel and I guess you'd have to set the io_offset value of that bus to 0x21530000 in order to make Linux I/O port 0 go to the first byte of the window and come out as 0xDEAD0000 on the bus, but you still won't be able to use legacy devices with hardcoded I/O port numbers.
Quoted 3 lines
If it is
0x00000000then the mmap scheme I outlined before must be used, and verify that only 0->0xFFFF is written to the PCI-PCI bridge IO base/limit registers..
For the primary bus, yes, but there are still two options for the second one: you can either start at 0 again or you can continue at 0x10000 as we do for mv78xx0 and kirkwood for instance. Both approaches probably have their merit.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Arnd Bergmann
Quoted 4 lines
If it is
0xDEAD0000, then Thomas has to keep what he has now, you can't mess with this address. Verify that the full 32 bit address exactly matching the MBUS window address is written to the PCI-PCI bridge IO base/limit registers.If you do this, you break all sorts of expectations in the kernel and I guess you'd have to set the
io_offsetvalue of that bus to0x21530000in order to make Linux I/O port 0 go to the first byte of the window and come out as0xDEAD0000on the bus, but you still won't be able to use legacy devices with hardcoded I/O port numbers.
I'm not sure exactly how the PCI core handles this, but it does look like pci_add_resource_offset via io_offset is the answer. I'm not sure what goes in the struct resource passed to the PCI core - the *bus* IO address range or the *kernel* IO address range..
Quoted 6 lines
Quoted 3 lines
If it is
0x00000000then the mmap scheme I outlined before must be used, and verify that only 0->0xFFFF is written to the PCI-PCI bridge IO base/limit registers..For the primary bus, yes, but there are still two options for the second one: you can either start at 0 again or you can continue
No, for *all* links. You use a mmap scheme with 4k granularity, I explained in a past email, but to quickly review..
- Each link gets 64k of reserved physical address space for IO, this is just set aside, no MBUS windows are permantently assigned.
- Linux is told to use a 64k IO range with bus IO address 0->0xFFFF
- When the IO base/limit register in the link PCI-PCI bridge is programmed the driver gets a 4k aligned region somewhere from 0->0xFFFF and then:
- Allocates a 64k MBUS window that translates physical address 0xZZZZxxxx to IO bus address 0x0000xxxx (goes in the TLP) for that link
- Uses pci_ioremap_io to map the fraction of the link's 64k MBUS window allocated to that bridge to the correct offset in the PCI_IO_VIRT_BASE region
So you'd end up with a MMU mapping something like:
PCI_IO_VIRT_BASE MBUS_IO_PHYS_BASE
0->4k => 0 -> 4k // 4k assigned to link0
4k->8k => 64k+4k -> 64k+8k // 4k assigned to link1
8k->24k => 128k+8k -> 128k+24k // 8k assigned to link2Where the physical mbus window for each link starts on each 64k block.
Thomas: This solves the need to have alignment of the IO regions, and gets rid of any trouble with 32 bit IO addreses, however you'll need to allocate the remap capable mbus windows separately for use by IO mappings..
Though, there is still a problem with the MMIO mbus window alignment. mbus windows are aligned to a multiple of their size, PCI MMIO bridge windows are always aligned to 1M...
Quoted 2 lines
at
0x10000as we do for mv78xx0 and kirkwood for instance. Both approaches probably have their merit.
Kirkwood uses the MBUS remapping registers to set the TLP address of link 0 to start at 0 and of link 1 to start at 0x10000 - so it is consistent with what you describe..
However, this is a suboptimal way to run the HW. It would be much better to place each link in a seperate PCI domain and have each link start its bus IO address at 0, and assign the kernel IO address in sequential 64k blocks as today.
Though, it is my hope that Thomas's driver will work on Kirkwood as well...
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 13 lines · Jason Gunthorpe
Quoted 10 lines · Arnd Bergmann
Quoted 4 lines
If it is
0xDEAD0000, then Thomas has to keep what he has now, you can't mess with this address. Verify that the full 32 bit address exactly matching the MBUS window address is written to the PCI-PCI bridge IO base/limit registers.If you do this, you break all sorts of expectations in the kernel and I guess you'd have to set the
io_offsetvalue of that bus to0x21530000in order to make Linux I/O port 0 go to the first byte of the window and come out as0xDEAD0000on the bus, but you still won't be able to use legacy devices with hardcoded I/O port numbers.
Quoted 4 lines
I'm not sure exactly how the PCI core handles this, but it does look like
pci_add_resource_offsetviaio_offsetis the answer. I'm not sure what goes in the struct resource passed to the PCI core - the *bus* IO address range or the *kernel* IO address range..
IO Resources are always expressed in the kernel's view, so they are in the range from 0 to IO_SPACE_LIMIT. The idea is that you can have multiple buses that each have their own address space start at 0, but can put them into the kernel address space at a different address.
Each device on any bus can still use I/O addresses starting at zero, and you could have e.g. a VGA card on two buses each respond to I/O cycles on port 0x3c0, but the PCI core will translate the resources to appear in the kernel space at 0x103c0 for the second one.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
If it is
0x00000000then the mmap scheme I outlined before must be used, and verify that only 0->0xFFFF is written to the PCI-PCI bridge IO base/limit registers..For the primary bus, yes, but there are still two options for the second one: you can either start at 0 again or you can continue
No, for *all* links. You use a mmap scheme with 4k granularity, I explained in a past email, but to quickly review..
Quoted 11 lines
- Each link gets 64k of reserved physical address space for IO, this is just set aside, no MBUS windows are permantently assigned.
- Linux is told to use a 64k IO range with bus IO address 0->0xFFFF
- When the IO base/limit register in the link PCI-PCI bridge is programmed the driver gets a 4k aligned region somewhere from 0->0xFFFF and then:
- Allocates a 64k MBUS window that translates physical address 0xZZZZxxxx to IO bus address 0x0000xxxx (goes in the TLP) for that link
- Usespci_ioremap_ioto map the fraction of the link's 64k MBUS window allocated to that bridge to the correct offset in thePCI_IO_VIRT_BASEregion
We'd have to change pci_ioremap_io to allow mapping less than 64k, but yes, that would work, too. I don't see an advantage to it though, other than having io_offset always be zero.
Quoted 6 lines
Quoted 2 lines
at
0x10000as we do for mv78xx0 and kirkwood for instance. Both approaches probably have their merit.Kirkwood uses the MBUS remapping registers to set the TLP address of link 0 to start at 0 and of link 1 to start at
0x10000- so it is consistent with what you describe..
Right, so it also uses io_offset = 0 all the time, which means the bus I/O port numbers are identical to the Linux I/O port numbers, but they go beyond 64K on the bus on the second and later links.
Quoted 4 lines
However, this is a suboptimal way to run the HW. It would be much better to place each link in a seperate PCI domain and have each link start its bus IO address at 0, and assign the kernel IO address in sequential 64k blocks as today.
I agree.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 5 lines · Arnd Bergmann
IO Resources are always expressed in the kernel's view, so they are in the range from 0 to
IO_SPACE_LIMIT. The idea is that you can have multiple buses that each have their own address space start at 0, but can put them into the kernel address space at a different address.
Sure, I see that.. and that seems reasonable as long as any IO bus address aliases are put in separate PCI domains. It would be wonky if devices on different PCI bus numbers in a single PCI domain had overlaping IO addresses.
Quoted 15 lines
Quoted 14 lines
No, for *all* links. You use a mmap scheme with 4k granularity, I explained in a past email, but to quickly review..
- Each link gets 64k of reserved physical address space for IO, this is just set aside, no MBUS windows are permantently assigned.
- Linux is told to use a 64k IO range with bus IO address 0->0xFFFF
- When the IO base/limit register in the link PCI-PCI bridge is programmed the driver gets a 4k aligned region somewhere from 0->0xFFFF and then:
- Allocates a 64k MBUS window that translates physical address 0xZZZZxxxx to IO bus address 0x0000xxxx (goes in the TLP) for that link
- Usespci_ioremap_ioto map the fraction of the link's 64k MBUS window allocated to that bridge to the correct offset in thePCI_IO_VIRT_BASEregion
Quoted 3 lines
We'd have to change
pci_ioremap_ioto allow mapping less than 64k, but yes, that would work, too. I don't see an advantage to it though, other than havingio_offsetalways be zero.
Erm, that is the whole point. No PCI device in the system, on any of the 10 links, would be required to use a 32 bit IO address. All are 16 bit and there is no compatibility problem on any links. You don't need to declare any one link as being 'io supporting' or something like that, it just works out of the box.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeDear Jason Gunthorpe,
Quoted 6 lines · Jason Gunthorpe
Quoted 2 lines
For the primary bus, yes, but there are still two options for the second one: you can either start at 0 again or you can continue
No, for *all* links. You use a mmap scheme with 4k granularity, I explained in a past email, but to quickly review..
Quoted 11 lines
- Each link gets 64k of reserved physical address space for IO, this is just set aside, no MBUS windows are permantently assigned.
- Linux is told to use a 64k IO range with bus IO address 0->0xFFFF
- When the IO base/limit register in the link PCI-PCI bridge is programmed the driver gets a 4k aligned region somewhere from 0->0xFFFF and then:
- Allocates a 64k MBUS window that translates physical address 0xZZZZxxxx to IO bus address 0x0000xxxx (goes in the TLP) for that link
- Usespci_ioremap_ioto map the fraction of the link's 64k MBUS window allocated to that bridge to the correct offset in thePCI_IO_VIRT_BASEregion
This, I think I now understand.
Quoted 5 lines
So you'd end up with a MMU mapping something like:
PCI_IO_VIRT_BASEMBUS_IO_PHYS_BASE0->4k => 0 -> 4k // 4k assigned to link0 4k->8k => 64k+4k -> 64k+8k // 4k assigned to link1 8k->24k => 128k+8k -> 128k+24k // 8k assigned to link2
I am not sure to understand your example, starting at the second line. Shouldn't the second line have been
4k->8k => 64k -> 64k+4k
?
If you do:
4k->8k => 64k+4k -> 64k+8k
as you suggested, then when the device driver will do an inl(0x4) on this device, the device will receive the equivalent of an inl(0x1004), no?
I understand that I have two choices here:
* First one is to make the I/O regions of all PCIe links fit below the default IO_SPACE_LIMIT (0xffff) by doing the mapping trick you described above.
* Second one is to have one 64 KB block for each PCIe link, which would require raising the IO_SPACE_LIMIT on this platform.
Is this correct?
If so, then what I don't understand is that Kirkwood does the second thing (from arch/arm/mach-kirkwood/pcie.c) :
switch (index) {
case 0:
[...]
/* Here the code is mapping 0 -> 64k */
pci_ioremap_io(SZ_64K * sys->busnr, KIRKWOOD_PCIE_IO_PHYS_BASE);
break;
case 1:
[...]
/* And here 64k -> 128k */
pci_ioremap_io(SZ_64K * sys->busnr,
KIRKWOOD_PCIE1_IO_PHYS_BASE);
break;So it has PCI I/O space from 0 to 128k, but still it seems to use the default IO_SPACE_LIMIT of 0xffff. How can this work? Maybe nobody every used a device on the second PCIe link that required I/O accesses.
Quoted 10 lines
Where the physical mbus window for each link starts on each 64k block.
Thomas: This solves the need to have alignment of the IO regions, and gets rid of any trouble with 32 bit IO addreses, however you'll need to allocate the remap capable mbus windows separately for use by IO mappings..
Though, there is still a problem with the MMIO mbus window alignment. mbus windows are aligned to a multiple of their size, PCI MMIO bridge windows are always aligned to 1M...
Can't this be solved using the window_alignement() hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?
Quoted 2 lines
Though, it is my hope that Thomas's driver will work on Kirkwood as well...
Yes, my plan is to have it working on Kirkwood. This WE, I was given a Kirkwood based machine that has a usable PCIe device.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 10 lines · Thomas Petazzoni
Quoted 5 lines
So you'd end up with a MMU mapping something like:
PCI_IO_VIRT_BASEMBUS_IO_PHYS_BASE0->4k => 0 -> 4k // 4k assigned to link0 4k->8k => 64k+4k -> 64k+8k // 4k assigned to link1 8k->24k => 128k+8k -> 128k+24k // 8k assigned to link2I am not sure to understand your example, starting at the second line. Shouldn't the second line have been
4k->8k => 64k -> 64k+4k
No..
Quoted 3 lines
as you suggested, then when the device driver will do an inl(0x4) on this device, the device will receive the equivalent of an inl(0x1004), no?
Link 0 translates like:
- Linux driver does inl(0x4)
- ARM layer converts that into a read from PCI_IO_VIRT_BASE + 0x4
- The CPU TLB converts that into a read from CPU physical 0xc0000000 + 0x4
- The MBUS window remap register converts that into a read from IO space 0x4
- The address 0x4 is placed in the PCI-E IO transaction of link 0
Link 1 translates like:
- Linux driver does inl(0x1004)
- ARM layer converts that into a read from PCI_IO_VIRT_BASE + 0x1004
- The CPU TLB converts that into a read from CPU physical 0xc0000000 + 0x11004 (ie the mbus window for the link 1)
- The MBUS window remap register converts that into a read from IO space 0x1004
- The address 0x1004 is placed in the PCI-E IO transaction of link 1
Noting that in both instances the IO address passed to inl is what eventually appears on the PCI-E link after all the translation is completed.
The CPU MMU is being used used to route 4k aligned ranges to the correct link.
Quoted 8 lines
I understand that I have two choices here:
* First one is to make the I/O regions of all PCIe links fit below the default
IO_SPACE_LIMIT(0xffff) by doing the mapping trick you described above.* Second one is to have one 64 KB block for each PCIe link, which would require raising the
IO_SPACE_LIMITon this platform.
Yes, however, AFIAK this is the environment you should be running in:
#define IO_SPACE_LIMIT ((resource_size_t)0xfffff)
Which is 5 f's not 4.
Quoted 8 lines
Quoted 3 lines
Though, there is still a problem with the MMIO mbus window alignment. mbus windows are aligned to a multiple of their size, PCI MMIO bridge windows are always aligned to 1M...
Can't this be solved using the
window_alignement()hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?
Hopefully :) As long as it can adjust the start and length you should be fine.
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Jason Gunthorpe,
Quoted 12 lines · Jason Gunthorpe
Link 0 translates like:
- Linux driver does inl(0x4)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x4
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x4
- The MBUS window remap register converts that into a read from IO space0x4
- The address0x4is placed in the PCI-E IO transaction of link 0Link 1 translates like:
Quoted 7 lines
- Linux driver does inl(0x1004)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x1004
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x11004(ie the mbus window for the link 1)
- The MBUS window remap register converts that into a read from IO space0x1004
- The address0x1004is placed in the PCI-E IO transaction of link 1
This last step is exactly what I thought would not work. If the PCIe device has say 64 bytes of I/O space, then this 0x1004 PCI-E transaction will be out of bounds, no?
Sorry, still learning how PCI works...
Quoted 6 lines
Quoted 6 lines
I understand that I have two choices here:
* First one is to make the I/O regions of all PCIe links fit below the default
IO_SPACE_LIMIT(0xffff) by doing the mapping trick you described above.
Quoted 8 lines
Quoted 2 lines
* Second one is to have one 64 KB block for each PCIe link, which would require raising the
IO_SPACE_LIMITon this platform.Yes, however, AFIAK this is the environment you should be running in:
#define
IO_SPACE_LIMIT((resource_size_t)0xfffff)Which is 5 f's not 4.
Aaah, you're right. My eyes got the number of f wrong. So I have 16 times 64 KB. So why would I bother doing the MMU trick if I can just nicely remap 64 KB for each PCIe link ?
Quoted 7 lines
Quoted 4 lines
Can't this be solved using the
window_alignement()hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?Hopefully :) As long as it can adjust the start and length you should be fine.
Why would you need to adjust the length? If Linux allocates a 2 MB resource on a 1 MB boundary, we simply increase the start address to the next 2 MB boundary, and that's it. Why would the length need to change?
Thanks,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 24 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
Quoted 12 lines · Jason Gunthorpe
Link 0 translates like:
- Linux driver does inl(0x4)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x4
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x4
- The MBUS window remap register converts that into a read from IO space0x4
- The address0x4is placed in the PCI-E IO transaction of link 0Link 1 translates like:
Quoted 7 lines
- Linux driver does inl(0x1004)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x1004
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x11004(ie the mbus window for the link 1)
- The MBUS window remap register converts that into a read from IO space0x1004
- The address0x1004is placed in the PCI-E IO transaction of link 1
Quoted 3 lines
This last step is exactly what I thought would not work. If the PCIe device has say 64 bytes of I/O space, then this
0x1004PCI-E transaction will be out of bounds, no?
No.. PCI end devices are required to decode all 32 bits of address, less the bits requires for their allocation. So a device with 64 bytes of IO will match bits 31:6 and then use bits 5:0 for the internal register.
So a full 32 bit address is technically fine, according to the spec, however:
- The 32 bit decode is an optional feature on bridges
- Some devices are broken because x86 only uses the low 64k.
So for best compatibility it is ideal to put everything in the low 64k.
However, IO space really should not be used by anything except very legacy devices, so if the MMU business is a hassle for some reason I'd just go with the 64k aligned method.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Can't this be solved using the
window_alignement()hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?
Quoted 7 lines
Quoted 2 lines
Hopefully :) As long as it can adjust the start and length you should be fine.
Why would you need to adjust the length? If Linux allocates a 2 MB resource on a 1 MB boundary, we simply increase the start address to the next 2 MB boundary, and that's it. Why would the length need to change?
Well, lets say 3MB is the example. A 3mb region needs to fit inside a 4mb MBUS window. If you align the start to 4mb then the pci-e core needs to know that it can't use the extra 1mb covered by the mbus window. mbus windows must not overlap.
Adjusting the bridge window length to be 4mb communicates that dead space to the PCI core, and presumably this shows up in lspci and whatnot.
I suppose if you align the end to 4mb (thus creating the gap before, not after) things should work out OK, but the information that the gap is routed to a specific PCI link is lost..
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Jason Gunthorpe,
Quoted 9 lines · Jason Gunthorpe
No.. PCI end devices are required to decode all 32 bits of address, less the bits requires for their allocation. So a device with 64 bytes of IO will match bits 31:6 and then use bits 5:0 for the internal register.
So a full 32 bit address is technically fine, according to the spec, however:
- The 32 bit decode is an optional feature on bridges
- Some devices are broken because x86 only uses the low 64k.
Thanks again for the great explanation, it is now clear to me.
Quoted 6 lines
So for best compatibility it is ideal to put everything in the low 64k.
However, IO space really should not be used by anything except very legacy devices, so if the MMU business is a hassle for some reason I'd just go with the 64k aligned method.
Me too :-)
It's what is done on Kirkwood since a long time, and apparently hasn't caused any trouble so far.
Quoted 9 lines
Quoted 4 lines
Why would you need to adjust the length? If Linux allocates a 2 MB resource on a 1 MB boundary, we simply increase the start address to the next 2 MB boundary, and that's it. Why would the length need to change?
Well, lets say 3MB is the example. A 3mb region needs to fit inside a 4mb MBUS window. If you align the start to 4mb then the pci-e core needs to know that it can't use the extra 1mb covered by the mbus window. mbus windows must not overlap.
Grumble, grumble, you're right. I now understand. Need to think of it, because the current pcibios_window_alignment() thing only allows to adjust the start address if I'm correct. But I'll have a more detailed look. Or maybe Russell can comment on this specific topic?
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Jason GunthorpeQuoted 25 lines · Jason Gunthorpe
Quoted 24 lines · Thomas Petazzoni
Dear Jason Gunthorpe,
Quoted 19 lines · Jason Gunthorpe
Link 0 translates like:
- Linux driver does inl(0x4)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x4
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x4
- The MBUS window remap register converts that into a read from IO space0x4
- The address0x4is placed in the PCI-E IO transaction of link 0Link 1 translates like:
- Linux driver does inl(0x1004)
- ARM layer converts that into a read fromPCI_IO_VIRT_BASE+0x1004
- The CPU TLB converts that into a read from CPU physical0xc0000000+0x11004(ie the mbus window for the link 1)
- The MBUS window remap register converts that into a read from IO space0x1004
- The address0x1004is placed in the PCI-E IO transaction of link 1
Quoted 8 lines
Quoted 3 lines
This last step is exactly what I thought would not work. If the PCIe device has say 64 bytes of I/O space, then this
0x1004PCI-E transaction will be out of bounds, no?No.. PCI end devices are required to decode all 32 bits of address, less the bits requires for their allocation. So a device with 64 bytes of IO will match bits 31:6 and then use bits 5:0 for the internal register.
Didn't Arnd say (earlier this thread) that PCI devices using IO BARs were probably fairly legacy and hence might be buggy and might not obey that rule? Now, I'd guess it's safe within the first 64k of IO space though, so perhaps he was only talking about IO BAR bases >= 64k being dubious? That would imply a device might only use bits 15:6 for matching the BAR base and 5:0 for the internal register for a 64-byte BAR.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 11 lines · Stephen Warren
Quoted 4 lines
No.. PCI end devices are required to decode all 32 bits of address, less the bits requires for their allocation. So a device with 64 bytes of IO will match bits 31:6 and then use bits 5:0 for the internal register.
Didn't Arnd say (earlier this thread) that PCI devices using IO BARs were probably fairly legacy and hence might be buggy and might not obey that rule? Now, I'd guess it's safe within the first 64k of IO space though, so perhaps he was only talking about IO BAR bases >= 64k being dubious? That would imply a device might only use bits 15:6 for matching the BAR base and 5:0 for the internal register for a 64-byte BAR.
Quoted 1 line · that is what I was referring to when I
- Some devices are broken because x86 only uses the low 64k.
Fortunately on PCI-E IO TLPs will be fully routed before they are sent down a link, so downstream of a PCI-E link we will never see aliasing of the low 16 bits.
This means if you do bridge to legacy PCI, and you do use devices that don't decode the upper 16 bits that it will still work OK, because the low 16 bits on the legacy PCI bus will still be unique in each device on that bus.
That doesn't save you from weird legacy ISA stuff, or stuff that doesn't respect the BARs, or other crazyness..
My personal hope would be that nobody using a PCI-E ARM SOC ever has to deal with anything to do with IO space ;)
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Stephen WarrenDear Stephen Warren,
Quoted 11 lines · Stephen Warren
Quoted 4 lines
No.. PCI end devices are required to decode all 32 bits of address, less the bits requires for their allocation. So a device with 64 bytes of IO will match bits 31:6 and then use bits 5:0 for the internal register.
Didn't Arnd say (earlier this thread) that PCI devices using IO BARs were probably fairly legacy and hence might be buggy and might not obey that rule? Now, I'd guess it's safe within the first 64k of IO space though, so perhaps he was only talking about IO BAR bases >= 64k being dubious? That would imply a device might only use bits 15:6 for matching the BAR base and 5:0 for the internal register for a 64-byte BAR.
The thing is that the existing PCIe support for earlier Marvell SoC families already use more than the first 64 KB to map the I/O BARs, and this hasn't apparently caused any problems. We're talking about PCIe support, not PCI, so I guess a lot of the very legacy devices are simply not part of the equation.
Can't we simply agree on having a first implementation that does the simple thing, like the existing PCIe implementation for earlier Marvell SoC families, and improve that if it happens to be needed, depending on user feedback?
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Thomas Petazzoni
Can't we simply agree on having a first implementation that does the simple thing, like the existing PCIe implementation for earlier Marvell SoC families, and improve that if it happens to be needed, depending on user feedback?
Makes sense. I just looked up the kirkwood source to verify that the window is set up to map PCI IO address 0x10000-0x1ffff for the second bus to KIRKWOOD_PCIE1_IO_PHYS_BASE, which is mapped to logical port number 0x10000-0x1ffff (identity mapping).
Arnd
Giving special alignment/size constraints to the Linux PCI core?
In reply to Jason GunthorpeBjorn,
I guess I need your advice on the below problem, but here is a summary.
Basically, the PCI-to-PCI bridge specifications require that the memory address assigned to a PCI-to-PCI bridge is aligned on a 1 MB boundary.
Unfortunately, on Marvell hardware, we can only create address decoding windows that are aligned on a multiple of their size (a 1 MB window must be 1 MB aligned, a 2 MB window must be 2 MB aligned, etc.).
How can we teach the Linux PCI core about this requirement, so that it does a proper assignment of addresses at the PCI-to-PCI bridge level? For the I/O addresses, Russell suggested the pcibios_window_alignment() hook, but it doesn't receive the size of the resource, so we can't determine what alignment is needed.
As Jason points out below, we need to be able to tell the PCI core that a given memory area needs some alignment, but also that its size is larger than what the PCIe device claims, because we cannot create address decoding windows of an arbitrary size. For example, an address decoding window of 3 MB is not possible, so if a device wants 3 MB, then we would need to extend this memory area to 4 MB so that the next device doesn't get an address decoding window that overlaps with the previous one.
I was hoping that the emulated PCI-to-PCI bridge could, by its behavior, teach the Linux PCI core about these special constraints. However, reading the PCI-to-PCI bridge specification, I don't see how to achieve that.
Do you have some suggestions?
I am unfortunately starting to believe that using the standard PCI resource allocator is too complicated for our hardware, and that we should maybe have a dedicated allocator. But I would really like to avoid that if possible.
Thanks a lot,
Thomas
Quoted 33 lines · Jason Gunthorpe
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Can't this be solved using the
window_alignement()hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?Quoted 7 lines
Quoted 2 lines
Hopefully :) As long as it can adjust the start and length you should be fine.
Why would you need to adjust the length? If Linux allocates a 2 MB resource on a 1 MB boundary, we simply increase the start address to the next 2 MB boundary, and that's it. Why would the length need to change?
Well, lets say 3MB is the example. A 3mb region needs to fit inside a 4mb MBUS window. If you align the start to 4mb then the pci-e core needs to know that it can't use the extra 1mb covered by the mbus window. mbus windows must not overlap.
Adjusting the bridge window length to be 4mb communicates that dead space to the PCI core, and presumably this shows up in lspci and whatnot.
I suppose if you align the end to 4mb (thus creating the gap before, not after) things should work out OK, but the information that the gap is routed to a specific PCI link is lost..
Jason
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Giving special alignment/size constraints to the Linux PCI core?
Quoted 4 lines · Thomas Petazzoni
I am unfortunately starting to believe that using the standard PCI resource allocator is too complicated for our hardware, and that we should maybe have a dedicated allocator. But I would really like to avoid that if possible.
I see this just as more evidence that the emulated P2P bridge approach is not the easiest solution and that it would be easier to go back to adding the ports separately and make it possible to have every port assign the BARs first and then set the resources based on the physical address of the window we pick for it.
Arnd
Bjorn Helgaas
maintainerGiving special alignment/size constraints to the Linux PCI core?
In reply to Thomas PetazzoniQuoted 26 lines · Thomas Petazzoni
Bjorn,
I guess I need your advice on the below problem, but here is a summary.
Basically, the PCI-to-PCI bridge specifications require that the memory address assigned to a PCI-to-PCI bridge is aligned on a 1 MB boundary.
Unfortunately, on Marvell hardware, we can only create address decoding windows that are aligned on a multiple of their size (a 1 MB window must be 1 MB aligned, a 2 MB window must be 2 MB aligned, etc.).
How can we teach the Linux PCI core about this requirement, so that it does a proper assignment of addresses at the PCI-to-PCI bridge level? For the I/O addresses, Russell suggested the
pcibios_window_alignment()hook, but it doesn't receive the size of the resource, so we can't determine what alignment is needed.As Jason points out below, we need to be able to tell the PCI core that a given memory area needs some alignment, but also that its size is larger than what the PCIe device claims, because we cannot create address decoding windows of an arbitrary size. For example, an address decoding window of 3 MB is not possible, so if a device wants 3 MB, then we would need to extend this memory area to 4 MB so that the next device doesn't get an address decoding window that overlaps with the previous one.
Quoted 6 lines
I was hoping that the emulated PCI-to-PCI bridge could, by its behavior, teach the Linux PCI core about these special constraints. However, reading the PCI-to-PCI bridge specification, I don't see how to achieve that.
Do you have some suggestions?
Huh. That hardware looks less and less like a P2P bridge all the time :( You can't configure it via standard PCI config accesses, and the aperture alignment and size constraints sound completely non-standard.
Are the specs for this thing public?
I could imagine changing pcibios_window_alignment() to take the resource, so it could deal with the alignment question (though I haven't looked in detail and there might be some implementation issue with that).
With regard to the size issue (3MB window using 4MB of address space), I can't think of a reasonable way to teach the PCI core about both sizes. But is there any reason to program the bridge for a 3MB window instead of a 4MB window, given that there's nothing else we can do with the extra 1MB anyway? Is a 3MB window even possible? I would think something that must be aligned on its size would be restricted to power-of-2 sizes anyway, just like PCI BARs are. Maybe you can just always round up window sizes to a power of 2?
Bjorn
Quoted 43 lines · Jason Gunthorpe
Quoted 28 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Can't this be solved using the
window_alignement()hook we've been discussing separately? Just like we teach the Linux PCI core about our alignment requirements of 64K for the I/O regions, we could teach it about our alignment requirement on memory regions as well. No?Quoted 7 lines
Quoted 2 lines
Hopefully :) As long as it can adjust the start and length you should be fine.
Why would you need to adjust the length? If Linux allocates a 2 MB resource on a 1 MB boundary, we simply increase the start address to the next 2 MB boundary, and that's it. Why would the length need to change?
Well, lets say 3MB is the example. A 3mb region needs to fit inside a 4mb MBUS window. If you align the start to 4mb then the pci-e core needs to know that it can't use the extra 1mb covered by the mbus window. mbus windows must not overlap.
Adjusting the bridge window length to be 4mb communicates that dead space to the PCI core, and presumably this shows up in lspci and whatnot.
I suppose if you align the end to 4mb (thus creating the gap before, not after) things should work out OK, but the information that the gap is routed to a specific PCI link is lost..
Quoted 6 lines
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
Giving special alignment/size constraints to the Linux PCI core?
Dear Bjorn Helgaas,
Quoted 4 lines · Bjorn Helgaas
Huh. That hardware looks less and less like a P2P bridge all the time :( You can't configure it via standard PCI config accesses, and the aperture alignment and size constraints sound completely non-standard.
Are the specs for this thing public?
The specs for the Armada XP are not yet public, but the one for earlier Marvell SoC families are, and the PCIe stuff works basically the same. The main difference between the Kirkwood family and Armada XP is that Kirkwood had only 2 PCIe interfaces, so we could perfectly fine do a static allocation of address decoding windows (see the four PCIe windows in arch/arm/mach-kirkwood/addr-map.c:addr_map_info[]), while the Armada XP has 10 PCIe interfaces, which makes the static allocation solution unreasonable.
The list of publicly available specifications for Marvell EBU SoC is available at Documentation/arm/Marvell/README. For Kirkwood, I would recommend
http://www.marvell.com/embedded-processors/kirkwood/assets/FS_88F6180_9x_6281_OpenSource.pdf.
See chapter 2.3:
The PCI Express address decoding scheme restricts the address window to a size of 2^n, and to a start address that is aligned to the window size.
Quoted 5 lines
I could imagine changing
pcibios_window_alignment()to take the resource, so it could deal with the alignment question (though I haven't looked in detail and there might be some implementation issue with that).
Quoted 8 lines
With regard to the size issue (3MB window using 4MB of address space), I can't think of a reasonable way to teach the PCI core about both sizes. But is there any reason to program the bridge for a 3MB window instead of a 4MB window, given that there's nothing else we can do with the extra 1MB anyway? Is a 3MB window even possible? I would think something that must be aligned on its size would be restricted to power-of-2 sizes anyway, just like PCI BARs are. Maybe you can just always round up window sizes to a power of 2?
The window sizes are power of two sizes. I didn't realize that it was also the case for PCI BARs. Then there is no problem with the size I guess, and only a problem of alignment. Having the possibility to get the resource and return a fixed up start address would solve the problem I'd say.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Giving special alignment/size constraints to the Linux PCI core?
Quoted 7 lines · Thomas Petazzoni
Dear Bjorn Helgaas,
Quoted 3 lines · Bjorn Helgaas
Huh. That hardware looks less and less like a P2P bridge all the time :( You can't configure it via standard PCI config accesses, and the aperture alignment and size constraints sound completely non-standard.
Right.
Quoted 14 lines
Quoted 5 lines
I could imagine changing
pcibios_window_alignment()to take the resource, so it could deal with the alignment question (though I haven't looked in detail and there might be some implementation issue with that).Quoted 8 lines
With regard to the size issue (3MB window using 4MB of address space), I can't think of a reasonable way to teach the PCI core about both sizes. But is there any reason to program the bridge for a 3MB window instead of a 4MB window, given that there's nothing else we can do with the extra 1MB anyway? Is a 3MB window even possible? I would think something that must be aligned on its size would be restricted to power-of-2 sizes anyway, just like PCI BARs are. Maybe you can just always round up window sizes to a power of 2?
Quoted 5 lines
The window sizes are power of two sizes. I didn't realize that it was also the case for PCI BARs. Then there is no problem with the size I guess, and only a problem of alignment. Having the possibility to get the resource and return a fixed up start address would solve the problem I'd say.
I thought that only device BARs in PCI had natural alignment, while bridges don't.
I tried understanding the actual problem we have with the current procedure, which on today's kirkwood is rought implemented in ARM's pci_common_init() as follows:
for_each_root_bus() {
pci_scan_root_bus();
}
for_each_root_bus {
pci_bus_size_bridges();
pci_bus_assign_resources();
pci_enable_bridges();
pci_bus_add_devices();
}This is using hardcoded windows today, which are set up before calling pci_scan_root_bus(). With your change, there is only one root bus, and you intercept the pci_bus_assign_resources() stage in order to set up the hardware specific window configuration for each PCIe port of that root bus.
My feeling is that an easier solution would be to keep separate root buses for each port, which then behaves completely PCIe compliant, but add a hook in the procedure above to set up the address translation windows between the pci_bus_size_bridges() and the pci_bus_assign_resources() calls.
Arnd
Giving special alignment/size constraints to the Linux PCI core?
Quoted 8 lines · Arnd Bergmann
Quoted 5 lines
The window sizes are power of two sizes. I didn't realize that it was also the case for PCI BARs. Then there is no problem with the size I guess, and only a problem of alignment. Having the possibility to get the resource and return a fixed up start address would solve the problem I'd say.
I thought that only device BARs in PCI had natural alignment, while bridges don't.
Right
Quoted 5 lines
My feeling is that an easier solution would be to keep separate root buses for each port, which then behaves completely PCIe compliant, but add a hook in the procedure above to set up the address translation windows between the
pci_bus_size_bridges()and thepci_bus_assign_resources()calls.
This process is only done during driver initialization. How would you support PCI-E device hotplug (my systems rely on this)? Hotplug works today with the existing Marvell driver, however that relies on pre-allocated windows.
Jason
Giving special alignment/size constraints to the Linux PCI core?
Quoted 10 lines · Jason Gunthorpe
Quoted 5 lines
My feeling is that an easier solution would be to keep separate root buses for each port, which then behaves completely PCIe compliant, but add a hook in the procedure above to set up the address translation windows between the
pci_bus_size_bridges()and thepci_bus_assign_resources()calls.This process is only done during driver initialization. How would you support PCI-E device hotplug (my systems rely on this)? Hotplug works today with the existing Marvell driver, however that relies on pre-allocated windows.
I did not expect hotplug to work with either approach. How does it work with the existing driver? From my understanding, you still assign all the top-level P2P bridge resources at bootup, and only if that happens to have some space left before the next bridge, it would be possible to fit in a hotplug device.
Arnd
Giving special alignment/size constraints to the Linux PCI core?
Quoted 12 lines · Arnd Bergmann
Quoted 10 lines · Jason Gunthorpe
Quoted 5 lines
My feeling is that an easier solution would be to keep separate root buses for each port, which then behaves completely PCIe compliant, but add a hook in the procedure above to set up the address translation windows between the
pci_bus_size_bridges()and thepci_bus_assign_resources()calls.This process is only done during driver initialization. How would you support PCI-E device hotplug (my systems rely on this)? Hotplug works today with the existing Marvell driver, however that relies on pre-allocated windows.
Quoted 5 lines
I did not expect hotplug to work with either approach. How does it work with the existing driver? From my understanding, you still assign all the top-level P2P bridge resources at bootup, and only if that happens to have some space left before the next bridge, it would be possible to fit in a hotplug device.
PCI-E hotplug can be supported through the generic /sys/bus/pci/rescan mechanism, which forces a rediscovery/re-evaluation of all the buses in the system. The PCI core is smart enough to know what it can/can not reassign and can move the bridge windows around (IIRC most of the issues here are resolved these days?).
For PCI-E, the root port bridge can be placed anywhere in the host aperture, so as long as the host aperture isn't filled the core can allocate a memory region for the now active port.
Thomas's driver should support this as the regions and windows are all properly dynamic.
This is one reason why re-using the PCI core code is so desirable, it handles all these complexities.
Jason
Giving special alignment/size constraints to the Linux PCI core?
Quoted 6 lines · Jason Gunthorpe
PCI-E hotplug can be supported through the generic
/sys/bus/pci/rescanmechanism, which forces a rediscovery/re-evaluation of all the buses in the system. The PCI core is smart enough to know what it can/can not reassign and can move the bridge windows around (IIRC most of the issues here are resolved these days?).
Quoted 6 lines
For PCI-E, the root port bridge can be placed anywhere in the host aperture, so as long as the host aperture isn't filled the core can allocate a memory region for the now active port.
Thomas's driver should support this as the regions and windows are all properly dynamic.
Ah, so you only allow hotplugging into the root ports, but not behind additional bridges that have active devices on them, right?
I guess that is a common limitation for PCIe hotplugging.
Quoted 2 lines
This is one reason why re-using the PCI core code is so desirable, it handles all these complexities.
Ok, I'm starting to understand what the entire discussion is about ;)
Arnd
Giving special alignment/size constraints to the Linux PCI core?
Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · Jason Gunthorpe
PCI-E hotplug can be supported through the generic
/sys/bus/pci/rescanmechanism, which forces a rediscovery/re-evaluation of all the buses in the system. The PCI core is smart enough to know what it can/can not reassign and can move the bridge windows around (IIRC most of the issues here are resolved these days?).For PCI-E, the root port bridge can be placed anywhere in the host aperture, so as long as the host aperture isn't filled the core can allocate a memory region for the now active port.
Quoted 7 lines
Quoted 2 lines
Thomas's driver should support this as the regions and windows are all properly dynamic.
Ah, so you only allow hotplugging into the root ports, but not behind additional bridges that have active devices on them, right?
I guess that is a common limitation for PCIe hotplugging.
In all the cases I've worked with, it has been a root port hot plug, but I could imagine something like ExpressCard requiring a second bridge.
The standard answer is to leave appropriate gaps. My *guess* on this matter is that on x86 the gaps are left, as appropriate, by the boot firmware. Eg an ExpressCard slot will always have a window assigned to its bridge and Linux would typically not reassign it (or similar).
PCI core support for firmware-less embedded will someday need to do something similar, eg via a special DT attribute on hot plug capable ports.
Just to circle back on this whole thread - Thomas's solution is pretty good, it covers pretty much all the use cases. I think it is a good place to start, and as the firmware-less 'drivers/pci/host' concept develops the right support will eventually come, as everyone is now aware of the need to control the host bridge aperture from the core PCI code.
Jason
Giving special alignment/size constraints to the Linux PCI core?
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · Arnd Bergmann
Quoted 12 lines · Jason Gunthorpe
PCI-E hotplug can be supported through the generic
/sys/bus/pci/rescanmechanism, which forces a rediscovery/re-evaluation of all the buses in the system. The PCI core is smart enough to know what it can/can not reassign and can move the bridge windows around (IIRC most of the issues here are resolved these days?).For PCI-E, the root port bridge can be placed anywhere in the host aperture, so as long as the host aperture isn't filled the core can allocate a memory region for the now active port.
Thomas's driver should support this as the regions and windows are all properly dynamic.
Quoted 8 lines
Quoted 4 lines
Ah, so you only allow hotplugging into the root ports, but not behind additional bridges that have active devices on them, right?
I guess that is a common limitation for PCIe hotplugging.
In all the cases I've worked with, it has been a root port hot plug, but I could imagine something like ExpressCard requiring a second bridge.
Ok, I see.
Quoted 8 lines
The standard answer is to leave appropriate gaps. My *guess* on this matter is that on x86 the gaps are left, as appropriate, by the boot firmware. Eg an ExpressCard slot will always have a window assigned to its bridge and Linux would typically not reassign it (or similar).
PCI core support for firmware-less embedded will someday need to do something similar, eg via a special DT attribute on hot plug capable ports.
I saw that the PCI core reserves 2MB memory space and 256 bytes of I/O space per hotplug capable bridge by default, and you can override these at boot time if you need more. I wonder if this means that we end up using two of the precious address space windows for each unused root port to already map these at boot time, and it certainly works for most adapters, but this does not seem better than assigning static windows of the same size at boot time for each port.
Quoted 6 lines
Just to circle back on this whole thread - Thomas's solution is pretty good, it covers pretty much all the use cases. I think it is a good place to start, and as the firmware-less '
drivers/pci/host' concept develops the right support will eventually come, as everyone is now aware of the need to control the host bridge aperture from the core PCI code.
I agree the solution is not all that bad, I just want to be convinced that it actually has advantages over the simpler approaches.
Arnd
Giving special alignment/size constraints to the Linux PCI core?
Quoted 5 lines · Arnd Bergmann
Quoted 5 lines
The standard answer is to leave appropriate gaps. My *guess* on this matter is that on x86 the gaps are left, as appropriate, by the boot firmware. Eg an ExpressCard slot will always have a window assigned to its bridge and Linux would typically not reassign it (or similar).
Quoted 7 lines
Quoted 3 lines
PCI core support for firmware-less embedded will someday need to do something similar, eg via a special DT attribute on hot plug capable ports.
I saw that the PCI core reserves 2MB memory space and 256 bytes of I/O space per hotplug capable bridge by default, and you can override
Haven't looked at how it determines what is hot plug capable.. Technically every PCI-E port is hot plug capable, it really depends on the specific board if a port can actually be hot plugged or not - so maybe that is what gets set in DT?
Quoted 5 lines
these at boot time if you need more. I wonder if this means that we end up using two of the precious address space windows for each unused root port to already map these at boot time, and it certainly works for most adapters, but this does not seem better than assigning static windows of the same size at boot time for each port.
If the PCI core programs the decoder on the bridge, then it will consume a window - however if there is nothing behind the bridge then leaving the brdige window disabled, but reserving the memory region is a sensible thing to do.
I'm not sure what the state of the PCI core is today on this point, but it could be altered..
Also the host driver can check the link status before consuming a window, no link = no window.
Thomas, what were your test results on your 10 slot system? Did all 10 P2P bridges appear in lspci? Was there any address space reservation for hot plug?
Quoted 9 lines
Quoted 6 lines
Just to circle back on this whole thread - Thomas's solution is pretty good, it covers pretty much all the use cases. I think it is a good place to start, and as the firmware-less '
drivers/pci/host' concept develops the right support will eventually come, as everyone is now aware of the need to control the host bridge aperture from the core PCI code.I agree the solution is not all that bad, I just want to be convinced that it actually has advantages over the simpler approaches.
Unfortunatelly my Marvell systems do not have oversubscribed mbus windows, so I can't really comment on this :( However I do use the hotplug capability in the current driver, so at least for me, it is important to not loose that when trying to solve the oversubcription.
Jason
Giving special alignment/size constraints to the Linux PCI core?
Dear Jason Gunthorpe,
Quoted 3 lines · Jason Gunthorpe
Thomas, what were your test results on your 10 slot system? Did all 10 P2P bridges appear in lspci? Was there any address space reservation for hot plug?
My test system "only" has 6 PCIe slots, so I couldn't test with the entire 10 possible PCIe interfaces. But 6 PCIe slots is still a good number :-)
If you look at the cover letter of the PATCHv3, you'll see the output of lspci -vvv. It shows that I create one PCI-to-PCI bridge for each PCIe interface, regardless of whether the link is here or not. And for those where there is nothing behind the bridge, no address space reservation occurs, so I am not allocating address decoding windows for those unused PCIe interfaces.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Giving special alignment/size constraints to the Linux PCI core?
In reply to Jason GunthorpeQuoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Arnd Bergmann
Quoted 8 lines
The standard answer is to leave appropriate gaps. My *guess* on this matter is that on x86 the gaps are left, as appropriate, by the boot firmware. Eg an ExpressCard slot will always have a window assigned to its bridge and Linux would typically not reassign it (or similar).
PCI core support for firmware-less embedded will someday need to do something similar, eg via a special DT attribute on hot plug capable ports.
Quoted 8 lines
Quoted 3 lines
I saw that the PCI core reserves 2MB memory space and 256 bytes of I/O space per hotplug capable bridge by default, and you can override
Haven't looked at how it determines what is hot plug capable.. Technically every PCI-E port is hot plug capable, it really depends on the specific board if a port can actually be hot plugged or not - so maybe that is what gets set in DT?
The "is_hotplug_bridge" flag that determines this gets set for PCIe bridges with the PCI_EXP_SLTCAP_HPC (hot plug capable) bit set in the PCI_EXP_SLTCAP word.
Quoted 6 lines
Quoted 5 lines
these at boot time if you need more. I wonder if this means that we end up using two of the precious address space windows for each unused root port to already map these at boot time, and it certainly works for most adapters, but this does not seem better than assigning static windows of the same size at boot time for each port.
Quoted 7 lines
If the PCI core programs the decoder on the bridge, then it will consume a window - however if there is nothing behind the bridge then leaving the brdige window disabled, but reserving the memory region is a sensible thing to do.
I'm not sure what the state of the PCI core is today on this point, but it could be altered..
The problem I see with the current implementation is that it reserves a fixed size window and does not reassign the window of the bridge itself, only the devices below it, at least if I am reading the code correctly. I have not tried this myself.
Quoted 2 lines
Also the host driver can check the link status before consuming a window, no link = no window.
Right, that works. Even if the link is up, it might require only I/O or memory windows, rather than always using both.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Just to circle back on this whole thread - Thomas's solution is pretty good, it covers pretty much all the use cases. I think it is a good place to start, and as the firmware-less '
drivers/pci/host' concept develops the right support will eventually come, as everyone is now aware of the need to control the host bridge aperture from the core PCI code.
Quoted 7 lines
Quoted 2 lines
I agree the solution is not all that bad, I just want to be convinced that it actually has advantages over the simpler approaches.
Unfortunatelly my Marvell systems do not have oversubscribed mbus windows, so I can't really comment on this :( However I do use the hotplug capability in the current driver, so at least for me, it is important to not loose that when trying to solve the oversubcription.
One thing worth trying is probably to hack the driver to only use a couple of the available windows and see what happens when you hotplug one card into all the slots one at a time.
Arnd
Giving special alignment/size constraints to the Linux PCI core?
Quoted 5 lines · Arnd Bergmann
The problem I see with the current implementation is that it reserves a fixed size window and does not reassign the window of the bridge itself, only the devices below it, at least if I am reading the code correctly. I have not tried this myself.
Which file?
Current code we *do* change the bridge itself resource (mmio, pref mmio, and io) in
pciehp_configure_device/pci_assign_unassigned_bridge_resources
Let met if it does not work you.
Thanks
Yinghai
Giving special alignment/size constraints to the Linux PCI core?
Quoted 7 lines · Yinghai Lu
Quoted 5 lines · Arnd Bergmann
The problem I see with the current implementation is that it reserves a fixed size window and does not reassign the window of the bridge itself, only the devices below it, at least if I am reading the code correctly. I have not tried this myself.
Quoted 7 lines
Which file?
Current code we do change the bridge itself resource (mmio, pref mmio, and io) in
pciehp_configure_device/pci_assign_unassigned_bridge_resourcesLet met if it does not work you.
That is the code I was looking at, but I probably misunderstood something there. I did not actually run it, just attempted to understand what it does by inspection.
I found the code now, sorry for the confusion on my end.
Jason, Thomas: you win :-)
The concept that root ports don't get resized is hardwired a lot of places along the way. That could be changed, but there is a significant risk of regressions if we try that. Adding fake bridges to work around that isn't the nicest solution, but the code is there and works without being able to break something else, so let's do that unless there are new problems that make it harder.
Arnd
Giving special alignment/size constraints to the Linux PCI core?
In reply to Jason GunthorpeQuoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Arnd Bergmann
Ah, so you only allow hotplugging into the root ports, but not behind additional bridges that have active devices on them, right?
I guess that is a common limitation for PCIe hotplugging.
Quoted 8 lines
In all the cases I've worked with, it has been a root port hot plug, but I could imagine something like ExpressCard requiring a second bridge.
The standard answer is to leave appropriate gaps. My *guess* on this matter is that on x86 the gaps are left, as appropriate, by the boot firmware. Eg an ExpressCard slot will always have a window assigned to its bridge and Linux would typically not reassign it (or similar).
Yes, x86 BIOS will leave some pad number for every pcie root port.
Also with patches ingit://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-pci-busn-alloc
could realloc bus range to get big bus number.
Thanks
Yinghai
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Russell King - ARM LinuxQuoted 65 lines · Russell King - ARM Linux
Quoted 31 lines · Russell King - ARM Linux
Quoted 30 lines · Thierry Reding
Quoted 28 lines · Russell King - ARM Linux
Quoted 5 lines · Thomas Petazzoni
(quoted diff) +5
-0+static resource_size_t mvebu_pcie_align_resource(struct pci_dev *dev, + const struct resource *res, + resource_size_t start, + resource_size_t size, + resource_size_t align)Review
Quoted 10 lines
(quoted diff) +10
-0+{ + if (!(res->flags & IORESOURCE_IO)) + return start; + + /* + * The I/O regions must be 64K aligned, because the + * granularity of PCIe I/O address decoding windows is 64 K + */ + return round_up(start, SZ_64K); +}You do realise that this will result in all PCI I/O BARs being rounded up to 64K.
I've just been digging through the PCI code and have come across a function -
pcibios_window_alignment()- which the PCI code allows to be overriden which allows you to increase the alignment requirement of bridge windows. It takes the PCI bus and window type as arguments.I'd suggest using that, and checking whether the bus which is passed corresponds with a bus which gives you problems, so that you don't impose the 64K requirement on downstream bridges.
Quoted 5 lines
Quoted 2 lines
That approach isn't going to work very well with multi-platform, though, since the function can only be overridden on a per-architecture basis.
The same can be said of all the various other functions which the PCI stuff expects the arch to provide, yet we seem to cope just fine...
And this (untested) is how it's done:
arch/arm/include/asm/mach/pci.h | 1 + arch/arm/kernel/bios32.c | 8 ++++++++ 2 files changed, 9 insertions(+), 0 deletions(-)arch/arm/include/asm/mach/pci.h +1
-0@@ -29,6 +29,7 @@ struct hw_pci {void (*postinit)(void );u8 (*swizzle)(struct pci_dev *dev,u8 *pin);int (*map_irq)(const struct pci_dev *dev,u8 slot,u8 pin); +resource_size_t (*window_align)(struct pci_bus *,unsigned long ); };/* arch/arm/kernel/bios32.c +1
-0@@ -581,6 +581,14 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,return start; } +resource_size_t pcibios_window_alignment(struct pci_bus *bus,
arch/arm/kernel/bios32.c resource_size_t pcibios_align_resource(void *data, const struct resource *res, · line 594
Quoted 10 lines
arch/arm/kernel/bios32.c +7
-0+ unsigned long type) +{ +struct pci_sys_data *sys = bus->sysdata; + +return sys->window_alignment ? sys->window_alignment (bus, type) :1 ; +} +/** * pcibios_enable_device - Enable I/O and memory. * @dev: PCI device to be enabled
Yes, something like that'll work. I had been under the impression that what you proposed was overriding pcibios_window_alignment() for Marvell only.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130131/88de3d77/attachment.sig>
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 4 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
Bjorn and I happen to live very close, so we got together today and talked about PCIe on ARM.
One of the questions he asked is: why does the window management on the Marvell SoCs need to be dynamic?
(Sorry if this was covered earlier; I vaguely recall some discussion on the topic, but couldn't find it quickly)
As background, PCIe enumeration in Linux usually works like:
1) You start off with some CPU physical address regions that generate transactions on the PCIe bus.
2) You enumerate all the PCIe devices, and assign an address to each BAR found, carved out of the PCIe address range corresponding to the regions you knew from (1).
However, it sounds like the Marvell code wants to:
1) Start off with no real knowledge of the CPU physical address that will generate transactions on the PCIe bus, since you want to assign that later.
2) You enumerate all the PCIe devices, and assign an address. But, what address range do you use?
3) Then you program the SoC's windows to set up the CPU->PCIe address translations.
Am I recalling what you're trying to do correctly, or am I completely confused?
Now, I recall that a related issue was that you are tight on CPU physical address space, and the second algorithm above would allow the size of the PCIe controller's window configuration to be as small as possible, and hence there would be more CPU physical address space available to fit in other peripherals.
However, why does this need to be dynamic? On a particular board, you know all the other (non-PCIe) peripherals that you need to fit into the CPU physical address space, so you know how much is left over for PCIe, so why not always make the PCIe window fill up all the available space, and use the first algorithm I described above? And also, I think you always know the exact set of PCIe devices that are attached to the boards, so you know the exact BAR size requirements there (or are there user-accessible PCIe devices; I don't think so from your recent comments about PCIe<->PCIe bridges not needing to be supported since the user couldn't plug one in?)
Note that with DT, you can easily specify the window location/size in the board .dts file rather than the SoC .dtsi file, so it can easily be customized based on how much physical address space is taken up by RAM, directly mapped NOR flash, etc.
With a static window configuration in DT, you'd end up with a system that worked much like any x86 system or Tegra, with some static memory range available to PCIe. It's just that in your case, the region location/size could change from boot to boot based on DT, whereas it's hard-coded in HW for Tegra and I assume x86 too.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 6 lines · Stephen Warren
Quoted 4 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
Quoted 8 lines
Bjorn and I happen to live very close, so we got together today and talked about PCIe on ARM.
One of the questions he asked is: why does the window management on the Marvell SoCs need to be dynamic?
(Sorry if this was covered earlier; I vaguely recall some discussion on the topic, but couldn't find it quickly)
Well I've answered it several times, so has Thomas.. Lets try again, please save for future reference :)
Lets seperate two things.
The CPU physical address ranges reserved for PCI bus access are not dynamic. This is set in DT, or whatever, statically. Just like every other PCI case on ARM. Just like Tegra. That is not the issue.
What is required is that the division of this space amongst the 10 physical PCI-E links must be dynamic. Just like x86. Just like the new tegra driver. [1]
Is that clear?
Quoted 8 lines
As background, PCIe enumeration in Linux usually works like:
1) You start off with some CPU physical address regions that generate transactions on the PCIe bus.
2) You enumerate all the PCIe devices, and assign an address to each BAR found, carved out of the PCIe address range corresponding to the regions you knew from (1).
Step 2 also includes 'assign address windows to all the physical PCI-E links'. This is very important because it is what this entire discussion is about.
Look at how tegra or x86 works, the CPU physical addresses for PCI-E do nothing until the PCI-to-PCI bridge window registers in each link's configuration space are setup. Until that is done the SOC doesn't know which link to send the transaction to.
Marvell is the same, until the link's window registers are setup the CPU addresses don't go anywhere.
Notice this has absolutely no effect on the host bridge aperture. This is a link-by-link configuration of what addresses *go down that link*.
The big difference is the link window registers for Marvell do not conform to the PCI configuration space specification. They are Marvell specific.
This is what the glue code in the host driver does, it converts the Marvell specificness into something the kernel can undertstand and control. There are countless ways to do this, but please accept it is necessary that it be done...
Quoted 5 lines
Now, I recall that a related issue was that you are tight on CPU physical address space, and the second algorithm above would allow the size of the PCIe controller's window configuration to be as small as possible, and hence there would be more CPU physical address space available to fit in other peripherals.
Physical address space is certainly a concern, but availability of decoder windows is the major one. Each link requires one decoder window for MMIO and one for IO, and possibly one for prefetch. The chip doesn't have 30 decoder windows. So the allocation of decoders to links must be dynamic, based on the requirements of the downstream endports on the link.
Quoted 5 lines
However, why does this need to be dynamic? On a particular board, you know all the other (non-PCIe) peripherals that you need to fit into the CPU physical address space, so you know how much is left over for PCIe, so why not always make the PCIe window fill up all the available space,
Because there is no such thing as an all-links PCIe window on this hardware.
Each link has a seperate window.
If you get rid of all the dynamic allocation then every link must statically reserve some portion of physical address space and some number of decoder windows.
That more or less means you need to know what is going to be on the other side of every link when you write the DT.
Quoted 5 lines
With a static window configuration in DT, you'd end up with a system that worked much like any x86 system or Tegra, with some static memory range available to PCIe. It's just that in your case, the region location/size could change from boot to boot based on DT, whereas it's hard-coded in HW for Tegra and I assume x86 too.
How is this better? Now you have a system where you have to customize the DT before you connect a PCI-E device. What if someone uses this chip in a configuration with physical slots? How does that work? What about hotplug? What about a unified kernel? That is *not* like x86 or tegra.
IMHO Thomas's direction in his proposed driver ends up working very close to the new tegra driver, and has the sort of dynamic allocation and discovery people expect from PCI-E.
Jason
1 - The new tegra driver switches from calling ARM's pci_common_init
once for every physical link, to once for the SOC. It does this by fixing the routing of config transactions so that the kernel sees the per-link PCI-PCI root port bridge config space provided by the hardware at the correct place. By doing this it changes from statically allocating a physical memory region for each link to statically allocating a region for all the links, and dynamically dividing that region amongst the links.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 25 lines · Jason Gunthorpe
Quoted 6 lines · Stephen Warren
Quoted 4 lines · Thomas Petazzoni
This driver implements the support for the PCIe interfaces on the Marvell Armada 370/XP ARM SoCs. In the future, it might be extended to cover earlier families of Marvell SoCs, such as Dove, Orion and Kirkwood.
Quoted 8 lines
Bjorn and I happen to live very close, so we got together today and talked about PCIe on ARM.
One of the questions he asked is: why does the window management on the Marvell SoCs need to be dynamic?
(Sorry if this was covered earlier; I vaguely recall some discussion on the topic, but couldn't find it quickly)
Well I've answered it several times, so has Thomas.. Lets try again, please save for future reference :)
Lets seperate two things.
The CPU physical address ranges reserved for PCI bus access are not dynamic. This is set in DT, or whatever, statically. Just like every other PCI case on ARM. Just like Tegra. That is not the issue.
Quoted 5 lines
What is required is that the division of this space amongst the 10 physical PCI-E links must be dynamic. Just like x86. Just like the new tegra driver. [1]
Is that clear?
Yes.
Quoted 5 lines
Quoted 5 lines
As background, PCIe enumeration in Linux usually works like:
1) You start off with some CPU physical address regions that generate transactions on the PCIe bus.
Quoted 7 lines
Quoted 3 lines
2) You enumerate all the PCIe devices, and assign an address to each BAR found, carved out of the PCIe address range corresponding to the regions you knew from (1).
Step 2 also includes 'assign address windows to all the physical PCI-E links'. This is very important because it is what this entire discussion is about.
OK.
Quoted 6 lines
Look at how tegra or x86 works, the CPU physical addresses for PCI-E do nothing until the PCI-to-PCI bridge window registers in each link's configuration space are setup. Until that is done the SOC doesn't know which link to send the transaction to.
From my perspective, this is slightly the wrong way of describing the
issue, but I see what you mean:
At least on Tegra and I think x86, any transaction that goes to the physical PCIe aperture is translated onto (internal) PCIe bus 0, so the lack of window or PCIe/PCIe bridge BAR register programming doesn't prevent the transaction going /somewhere/ (even if "somewhere" is only half way to where it's useful!). The difference is pretty subtle. The issue is that without the PCIe/PCIe bridge BARs programmed, the PCIe transactions won't get off bus 0 and onto a downstream bus of one of the PCIe/PCIe bridges, or put another way, no PCIe/PCIe will claim the transaction that happens on PCIe bus 0
(Using "PCIe/PCIe bridge" above to mean "PCIe root port")
Quoted 6 lines
Marvell is the same, until the link's window registers are setup the CPU addresses don't go anywhere.
Notice this has absolutely no effect on the host bridge aperture. This is a link-by-link configuration of what addresses *go down that link*.
Right.
Quoted 8 lines
The big difference is the link window registers for Marvell do not conform to the PCI configuration space specification. They are Marvell specific.
This is what the glue code in the host driver does, it converts the Marvell specificness into something the kernel can undertstand and control. There are countless ways to do this, but please accept it is necessary that it be done...
Sure.
Quoted 6 lines
Quoted 5 lines
Now, I recall that a related issue was that you are tight on CPU physical address space, and the second algorithm above would allow the size of the PCIe controller's window configuration to be as small as possible, and hence there would be more CPU physical address space available to fit in other peripherals.
Quoted 6 lines
Physical address space is certainly a concern, but availability of decoder windows is the major one. Each link requires one decoder window for MMIO and one for IO, and possibly one for prefetch. The chip doesn't have 30 decoder windows. So the allocation of decoders to links must be dynamic, based on the requirements of the downstream endports on the link.
Oh I see...
I originally thought the issue was that the windows were between CPU physical address space and the PCIe host controller itself. But in fact, the windows are between PCIe bus 0 and the root ports, so they're the equivalent of the standard PCIe root port (or PCIe/PCIe bridge) BAR registers. And related, these BAR/window registers are usually part of each PCIe root port itself, and hence there's a whole number dedicated to each root port, but on Marvell there's a *global* pool of these BARs/windows instead.
Now I think I finally understand the architecture of your HW.
Quoted 9 lines
Quoted 5 lines
However, why does this need to be dynamic? On a particular board, you know all the other (non-PCIe) peripherals that you need to fit into the CPU physical address space, so you know how much is left over for PCIe, so why not always make the PCIe window fill up all the available space,
Because there is no such thing as an all-links PCIe window on this hardware.
Quoted 8 lines
Each link has a seperate window.
If you get rid of all the dynamic allocation then every link must statically reserve some portion of physical address space and some number of decoder windows.
That more or less means you need to know what is going to be on the other side of every link when you write the DT.
So, the dynamic programming of the windows on Marvell HW is the exact logical equivalent of programming a standard PCIe root port's BAR registers. It makes perfect sense that should be dynamic. Presumably this is something you can make work inside your emulated PCIe/PCIe bridge module, simply by capturing writes to the BAR registers, and translating them into writes to the Marvell window registers.
Now, I do have one follow-on question: You said you don't have 30 windows, but how many do you have free after allocating windows to any other peripherals that need them, relative to (3 * number-of-root-ports-in-the-SoC)? (3 being IO+Mem+PrefetchableMem.)
The thing here is that when the PCIe core writes to a root port BAR window to configure/enable it the first time, you'll need to capture that transaction and dynamically allocate a window and program it in a way equivalent to what the BAR register write would have achieved on standard HW. Later, the window might need resizing, or even to be completely disabled, if the PCIe core were to change the standard BAR register. Dynamically allocating a window when the BAR is written seems a little heavy-weight.
So while it's obvious that window base address and size shouldn't be static, I wonder if the assignment of a specific window ID to a specific root port ID shouldn be dynamic or static. For example, if your HW configuration leaves you with 6 windows available, you could support 2 PCIe root ports by statically assigning 3 windows to serve each of those 2 root ports. Would that work, or are there systems where over-commit is needed, e.g. if there's no IO space behind a root port, you could get away with two windows per root port, and hence be able to run 3 root ports rather than just 2? Still, if you know which PCIe devices are being the root ports, you could still represent the over-commit statically in DT
Still, I supose doing it dynamically in the driver does end up being a lot less to think about for someone creating the DT for a new board.
Having to translate standard root port BAR register writes to Marvell window register allocation/writes would imply that the emulated root port code has to be very closely tied into the Marvell PCIe driver, and not something that could be at all generic in the most part.
Quoted 11 lines
Quoted 5 lines
With a static window configuration in DT, you'd end up with a system that worked much like any x86 system or Tegra, with some static memory range available to PCIe. It's just that in your case, the region location/size could change from boot to boot based on DT, whereas it's hard-coded in HW for Tegra and I assume x86 too.
How is this better? Now you have a system where you have to customize the DT before you connect a PCI-E device. What if someone uses this chip in a configuration with physical slots? How does that work? What about hotplug? What about a unified kernel? That is *not* like x86 or tegra.
Right. Now that I really understand what the windows are doing, I can see that a static window configuration (address/size, perhaps rather than windows are used) would not be appropriate.
Quoted 6 lines
IMHO Thomas's direction in his proposed driver ends up working very close to the new tegra driver, and has the sort of dynamic allocation and discovery people expect from PCI-E.
Jason
Quoted 8 lines
1 - The new tegra driver switches from calling ARM's
pci_common_init
once for every physical link, to once for the SOC. It does this by fixing the routing of config transactions so that the kernel sees the per-link PCI-PCI root port bridge config space provided by the hardware at the correct place. By doing this it changes from statically allocating a physical memory region for each link to statically allocating a region for all the links, and dynamically dividing that region amongst the links.
Right, we have both (or all 3) root ports show up in the same PCIe domain.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Stephen Warren
I originally thought the issue was that the windows were between CPU physical address space and the PCIe host controller itself. But in fact, the windows are between PCIe bus 0 and the root ports, so they're the
Donno what exactly is inside tegra, but on Marvell the docs describe an an internal bus cross bar/whatever and each PCI-E link gets a port on that structure. There is no such physical thing as a 'PCI bus 0', and they didn't arrange the hardware in a way that makes it easy for the host driver to create one like tegra did :(
Quoted 5 lines
equivalent of the standard PCIe root port (or PCIe/PCIe bridge) BAR registers. And related, these BAR/window registers are usually part of each PCIe root port itself, and hence there's a whole number dedicated to each root port, but on Marvell there's a *global* pool of these BARs/windows instead.
Right, that is all following the PCI-E spec, and is reasonable and sane. What Marvell did is take an *end port core* and jam'd it on their internal bus without adjusting things to follow the PCI-E spec regarding config space and what not.
Quoted 9 lines
Quoted 2 lines
That more or less means you need to know what is going to be on the other side of every link when you write the DT.
So, the dynamic programming of the windows on Marvell HW is the exact logical equivalent of programming a standard PCIe root port's BAR registers. It makes perfect sense that should be dynamic. Presumably this is something you can make work inside your emulated PCIe/PCIe bridge module, simply by capturing writes to the BAR registers, and translating them into writes to the Marvell window registers.
Yes, that is exactly the idea.
Quoted 4 lines
Now, I do have one follow-on question: You said you don't have 30 windows, but how many do you have free after allocating windows to any other peripherals that need them, relative to (3 * number-of-root-ports-in-the-SoC)? (3 being IO+Mem+PrefetchableMem.)
Thomas will have to answer this, it varies depending on the SOC, and what other on chip peripherals are in use. For instance Kirkwood has the same design but there are plenty of windows for the two PCI-E links.
Still how would you even connect a limited number of regions on a link by link basis to the common PCI code?
Quoted 7 lines
The thing here is that when the PCIe core writes to a root port BAR window to configure/enable it the first time, you'll need to capture that transaction and dynamically allocate a window and program it in a way equivalent to what the BAR register write would have achieved on standard HW. Later, the window might need resizing, or even to be completely disabled, if the PCIe core were to change the standard BAR
Right. This is pretty straightforward except for the need to hook the alignment fixup..
Quoted 2 lines
register. Dynamically allocating a window when the BAR is written seems a little heavy-weight.
I think what Thomas had here was pretty small, and the windows need to be shared with other on chip periphals beyond PCI-E..
Quoted 3 lines
needed, e.g. if there's no IO space behind a root port, you could get away with two windows per root port, and hence be able to run 3 root ports rather than just 2?
Right, this is the main point. If you plug in 3 devices and they all only use MMIO regions then you only need to grab 3 windows. The kernel disables the unused windows on the bridge so it is easy to tell when they are disused.
Quoted 2 lines
Still, I supose doing it dynamically in the driver does end up being a lot less to think about for someone creating the DT for a new board.
Agreed, these restrictions are all so HW specific, subtle and have nothing to do with the PCI-E spec. Codifying them once in the driver seems like the way to keep this crazyness out of the PCI core and away from users of the SOC.
Quoted 4 lines
Having to translate standard root port BAR register writes to Marvell window register allocation/writes would imply that the emulated root port code has to be very closely tied into the Marvell PCIe driver, and not something that could be at all generic in the most part.
Agreed.. At the very least generic code would need call back functions to the driver... It has a fair bit to do for Marvell:
- Translate MMIO, prefetch and IO ranges to mbus windows
- Keep track of the secondary/subordinate bus numbers and fiddle with other hardware registers to set those up
- Copy the link state/control regsiters from the end port config space into the bridge express root port capability
- Probably ditto for AER as well..
Probably simpler just to make one for marvell then mess excessively with callbacks..
Cheers,
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Dear Jason Gunthorpe,
Thanks again for continuing this discussion while I was sleeping :-)
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines
Now, I do have one follow-on question: You said you don't have 30 windows, but how many do you have free after allocating windows to any other peripherals that need them, relative to (3 * number-of-root-ports-in-the-SoC)? (3 being IO+Mem+PrefetchableMem.)
Thomas will have to answer this, it varies depending on the SOC, and what other on chip peripherals are in use. For instance Kirkwood has the same design but there are plenty of windows for the two PCI-E links.
Right. I already answered this point directly to Stephen. On Kirkwood, there are many windows and two PCIe links, so the windows were statically allocated. On Armada XP, there are 20 windows and 10 PCIe links. Static allocation is no longer reasonable.
Quoted 8 lines
Quoted 7 lines
The thing here is that when the PCIe core writes to a root port BAR window to configure/enable it the first time, you'll need to capture that transaction and dynamically allocate a window and program it in a way equivalent to what the BAR register write would have achieved on standard HW. Later, the window might need resizing, or even to be completely disabled, if the PCIe core were to change the standard BAR
Quoted 8 lines
Right. This is pretty straightforward except for the need to hook the alignment fixup..
Quoted 2 lines
register. Dynamically allocating a window when the BAR is written seems a little heavy-weight.
I think what Thomas had here was pretty small, and the windows need to be shared with other on chip periphals beyond PCI-E..
Yes, it is not very complicated. We already have some common code that creates/removes those windows, so it is just a matter of calling the right thing at the right time. Definitely not hundreds of line of crap.
Quoted 4 lines
Right, this is the main point. If you plug in 3 devices and they all only use MMIO regions then you only need to grab 3 windows. The kernel disables the unused windows on the bridge so it is easy to tell when they are disused.
Ah, I'm interested in further discussing this. I currently have a setup with one SATA PCIe card and one NIC PCIe card. On the NIC, the I/O ports are said to be "disabled", but still an I/O region gets allocated in the PCI-to-PCI bridge that gives access to this particular device.
The device in question is:
05:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 106
Region 0: Memory at c1200000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at c1220000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at c0010000 [disabled] [size=32]
[virtual] Expansion ROM@c1300000 [disabled] [size=128K]
So the Region 2 is disabled. But, in the corresponding bridge:
00:05.0 PCI bridge: Marvell Technology Group Ltd. Device 1092 (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: c0010000-c001ffff
Memory behind bridge: c1200000-c12fffff
Prefetchable memory behind bridge: c1300000-c13fffff
So there is really a range of I/O addresses associated to it, even though the device will apparently not use it. Would it be possible to detect that the I/O range is not used by the device, and therefore avoid the allocation of an address decoding window for this I/O range?
Quoted 11 lines
Agreed.. At the very least generic code would need call back functions to the driver... It has a fair bit to do for Marvell:
- Translate MMIO, prefetch and IO ranges to mbus windows
- Keep track of the secondary/subordinate bus numbers and fiddle with other hardware registers to set those up
- Copy the link state/control regsiters from the end port config space into the bridge express root port capability
- Probably ditto for AER as well..Probably simpler just to make one for marvell then mess excessively with callbacks..
As replied to Stephen, I've chosen to bring the PCI-to-PCI bridge emulation code directly into the driver, specifically for this reason.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 4 lines · Thomas Petazzoni
So there is really a range of I/O addresses associated to it, even though the device will apparently not use it. Would it be possible to detect that the I/O range is not used by the device, and therefore avoid the allocation of an address decoding window for this I/O range?
I suspect it just gets disabled because the port number 0xc0010000 is larger than IO_PORT_LIMIT and we cannot access that offset inside of the virtual memory window we use for PIO.
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines · Thomas Petazzoni
So there is really a range of I/O addresses associated to it, even though the device will apparently not use it. Would it be possible to detect that the I/O range is not used by the device, and therefore avoid the allocation of an address decoding window for this I/O range?
I suspect it just gets disabled because the port number
0xc0010000is larger thanIO_PORT_LIMITand we cannot access that offset inside of the virtual memory window we use for PIO.
You're running into that trap again which you fall into on other architectures.
If you arrange for your PCI IO space to start at 0 rather than the physical address that it appears on your CPU, then you shouldn't end up with it extending up to something at 0xcXXXXXXX.
Remember that we should be ensuring that inb(0) hits the first address of the cross-subarch PCI IO area - this alway requires that any sub-arch taking part in a multiplatform kernel must start its IO space addresses at 0 and not the physical address on the local CPU.
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 14 lines · Russell King - ARM Linux
Quoted 9 lines · Arnd Bergmann
Quoted 4 lines · Thomas Petazzoni
So there is really a range of I/O addresses associated to it, even though the device will apparently not use it. Would it be possible to detect that the I/O range is not used by the device, and therefore avoid the allocation of an address decoding window for this I/O range?
I suspect it just gets disabled because the port number
0xc0010000is larger thanIO_PORT_LIMITand we cannot access that offset inside of the virtual memory window we use for PIO.You're running into that trap again which you fall into on other architectures.
Quoted 8 lines
If you arrange for your PCI IO space to start at 0 rather than the physical address that it appears on your CPU, then you shouldn't end up with it extending up to something at 0xcXXXXXXX.
Remember that we should be ensuring that inb(0) hits the first address of the cross-subarch PCI IO area - this alway requires that any sub-arch taking part in a multiplatform kernel must start its IO space addresses at 0 and not the physical address on the local CPU.
Yes, that was my point. I think in this case, the bug is in the new of_pci_process_ranges functions, which returns a 'struct resource' translated into IORESOURCE_MEM space, but with the type set to IORESOURCE_IO. This resource then gets passed to pci_add_resource_offset().
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 5 lines · Arnd Bergmann
Yes, that was my point. I think in this case, the bug is in the new
of_pci_process_rangesfunctions, which returns a 'struct resource' translated intoIORESOURCE_MEMspace, but with the type set toIORESOURCE_IO. This resource then gets passed topci_add_resource_offset().
A standard way to express the required address translation from CPU physical address to IO bus address might help other people avoid this trap??
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Stephen WarrenDear Stephen Warren,
Thanks for this great discussion. I see that Jason has already answered most of the questions on the why we need this "dynamicity" in the window configuration. A few comments below.
Quoted 6 lines · Stephen Warren
So, the dynamic programming of the windows on Marvell HW is the exact logical equivalent of programming a standard PCIe root port's BAR registers. It makes perfect sense that should be dynamic. Presumably this is something you can make work inside your emulated PCIe/PCIe bridge module, simply by capturing writes to the BAR registers, and translating them into writes to the Marvell window registers.
That's what I'm doing. In the PATCHv2, my PCIe host driver was reading back the BARs in the PCI-to-PCI bridges configuration space, and was setting up the windows according to the addresses that had been assigned to each bridge.
I am currently working in making this more dynamic: it is directly when the BAR is being written to in the PCI-to-PCI bridge configuration space that a window will be setup.
Quoted 4 lines
Now, I do have one follow-on question: You said you don't have 30 windows, but how many do you have free after allocating windows to any other peripherals that need them, relative to (3 * number-of-root-ports-in-the-SoC)? (3 being IO+Mem+PrefetchableMem.)
We have 20 windows on Armada XP if I remember correctly, and they are not only used for PCIe, but also to map the BootROM (needed to boot secondary CPUs), to map SPI flashes or NOR flashes, for example. So they are really shared between many uses. In terms of PCIe, there are only two types of windows: I/O and Memory, there is no notion of Prefetchable Memory window as far as I could see.
We have up to 10 PCIe interfaces, and only 20 windows. It means that you basically can't use all PCIe interfaces, there will necessarily be some limit, due to the limited number of windows.
Also, I'd like to point out that the dynamic configuration is needed for two reasons:
* The number of windows, as we are discussing now.
* The amount of physical address space available. If you don't dynamically configure those windows, then you have to account the "worst case", i.e the PCIe devices that require very large memory areas. So you end up creating static windows that reserve 32M or 64M or 128M *per* PCIe link. You can see that it "consumes" pretty quickly a large part of the 4G physical address space that we have. Thanks to the dynamic window configuration that we do with the PCI-to-PCI bridge, we can size the windows exactly the size needed by the downstream device on each PCIe interface.
Quoted 8 lines
The thing here is that when the PCIe core writes to a root port BAR window to configure/enable it the first time, you'll need to capture that transaction and dynamically allocate a window and program it in a way equivalent to what the BAR register write would have achieved on standard HW. Later, the window might need resizing, or even to be completely disabled, if the PCIe core were to change the standard BAR register. Dynamically allocating a window when the BAR is written seems a little heavy-weight.
Why?
Quoted 11 lines
So while it's obvious that window base address and size shouldn't be static, I wonder if the assignment of a specific window ID to a specific root port ID shouldn be dynamic or static. For example, if your HW configuration leaves you with 6 windows available, you could support 2 PCIe root ports by statically assigning 3 windows to serve each of those 2 root ports. Would that work, or are there systems where over-commit is needed, e.g. if there's no IO space behind a root port, you could get away with two windows per root port, and hence be able to run 3 root ports rather than just 2? Still, if you know which PCIe devices are being the root ports, you could still represent the over-commit statically in DT
For now, I haven't figured out how not to allocate an I/O window if the downstream device doesn't use I/O, but I'd like to achieve that, as it would save one of the two windows needed per PCIe interface... and many PCIe devices don't need the I/O window.
Quoted 2 lines
Still, I supose doing it dynamically in the driver does end up being a lot less to think about for someone creating the DT for a new board.
Indeed.
Quoted 4 lines
Having to translate standard root port BAR register writes to Marvell window register allocation/writes would imply that the emulated root port code has to be very closely tied into the Marvell PCIe driver, and not something that could be at all generic in the most part.
Right. I've already moved the PCI-to-PCI bridge code from the generic drivers/pci/sw-pci-pci-bridge.c location to be directly into the driver. It also to integrate more tightly things like window allocation.
Quoted 3 lines
Right. Now that I really understand what the windows are doing, I can see that a static window configuration (address/size, perhaps rather than windows are used) would not be appropriate.
Glad to see we reached the same conclusion :-)
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 12 lines · Thomas Petazzoni
Quoted 11 lines
So while it's obvious that window base address and size shouldn't be static, I wonder if the assignment of a specific window ID to a specific root port ID shouldn be dynamic or static. For example, if your HW configuration leaves you with 6 windows available, you could support 2 PCIe root ports by statically assigning 3 windows to serve each of those 2 root ports. Would that work, or are there systems where over-commit is needed, e.g. if there's no IO space behind a root port, you could get away with two windows per root port, and hence be able to run 3 root ports rather than just 2? Still, if you know which PCIe devices are being the root ports, you could still represent the over-commit statically in DT
Quoted 4 lines
For now, I haven't figured out how not to allocate an I/O window if the downstream device doesn't use I/O, but I'd like to achieve that, as it would save one of the two windows needed per PCIe interface... and many PCIe devices don't need the I/O window.
The easiest hack would be to only ever allow one I/O window on exactly one of the ports, and not do PIO on the other ports at all. Given the various troubles of making any other combination work, that sounds like a good enough compromise to me. A lot of the add-on cards would not work in the remaining ports anyway, and worrying too much about that legacy hardware may just not be worth it. That way, you only need 11 windows for PCIe (10*mem, 1*IO), which will always fit.
Do you actually have /any/ PCIe cards with PIO BARs that you can test with?
Arnd
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
In reply to Thomas PetazzoniQuoted 29 lines · Thomas Petazzoni
Dear Stephen Warren,
Thanks for this great discussion. I see that Jason has already answered most of the questions on the why we need this "dynamicity" in the window configuration. A few comments below.
Quoted 6 lines · Stephen Warren
So, the dynamic programming of the windows on Marvell HW is the exact logical equivalent of programming a standard PCIe root port's BAR registers. It makes perfect sense that should be dynamic. Presumably this is something you can make work inside your emulated PCIe/PCIe bridge module, simply by capturing writes to the BAR registers, and translating them into writes to the Marvell window registers.
That's what I'm doing. In the PATCHv2, my PCIe host driver was reading back the BARs in the PCI-to-PCI bridges configuration space, and was setting up the windows according to the addresses that had been assigned to each bridge.
I am currently working in making this more dynamic: it is directly when the BAR is being written to in the PCI-to-PCI bridge configuration space that a window will be setup.
Quoted 4 lines
Now, I do have one follow-on question: You said you don't have 30 windows, but how many do you have free after allocating windows to any other peripherals that need them, relative to (3 * number-of-root-ports-in-the-SoC)? (3 being IO+Mem+PrefetchableMem.)
Quoted 6 lines
We have 20 windows on Armada XP if I remember correctly, and they are not only used for PCIe, but also to map the BootROM (needed to boot secondary CPUs), to map SPI flashes or NOR flashes, for example. So they are really shared between many uses. In terms of PCIe, there are only two types of windows: I/O and Memory, there is no notion of Prefetchable Memory window as far as I could see.
In Tegra, we end up having separate MMIO vs. Prefetchable MMIO chunks of our overall PCIe aperture. However, the HW setup appears the same for both of those. I'm not sure if it's a bug in the driver, or if it's just to separate the two address spaces so that the page tables can be configured for those two regions with large rather than small granularity. I need to go investigate that.
Quoted 3 lines
We have up to 10 PCIe interfaces, and only 20 windows. It means that you basically can't use all PCIe interfaces, there will necessarily be some limit, due to the limited number of windows.
So there are 10 PCIe interfaces (root ports). That's on the SoC itself right. Are all 10 (or a large number of them) actually used at once on any given board design? I suppose this must be the case, or Marvell wouldn't have wasted the silicon space on 10 root ports... Still, that's a rather large number of ports!
If only a few PCIe ports are ever in use at once on a design and/or the PCIe ports generally contain soldered-down devices rather than user-accessible ports, the statically assigning window *IDs* to individual ports would make for easier code in the driver, since the BAR register emulation would never have to allocate/de-allocate windows, but rather only ever have to enable/disable/configure them.
However, if many PCIe ports are in use at once and there are user-accessible ports, you can't know ahead of time which ports will need MMIO vs. MMIO prefetch vs. IO, so you'd have to dynamically allocate window IDs to ports, in addition to dynamically setting up the address/size of windows.
Quoted 4 lines
Also, I'd like to point out that the dynamic configuration is needed for two reasons:
* The number of windows, as we are discussing now.
OK.
Quoted 9 lines
* The amount of physical address space available. If you don't dynamically configure those windows, then you have to account the "worst case", i.e the PCIe devices that require very large memory areas. So you end up creating static windows that reserve 32M or 64M or 128M *per* PCIe link. You can see that it "consumes" pretty quickly a large part of the 4G physical address space that we have. Thanks to the dynamic window configuration that we do with the PCI-to-PCI bridge, we can size the windows exactly the size needed by the downstream device on each PCIe interface.
That aspect is applicable to any PCIe system; there's always some chunk of physical address space that maps to PCIe, and which must be divided into per-root-port chunks.
I think the only difference on the Marvell HW is:
* The overall total size of the physical address space is dynamic rather than fixed, because it's programmed through windows rather than hard-coded into HW.
* Hence, the windows /both/ define the physical address space layout /and/ define the routing of transactions to individual root ports. On regular PCIe, the root port BARs only divide up the overall physical (PCIe bus 0 really) address space and hence perform routing; they have no influence over the CPU physical address space.
So I think the crux of the problem is that you really have 10 PCIe root ports, each of which is a nominally a separate PCIe domain (since the windows connect CPU physical address space to an individual/specific root port's PCIe address space, rather than having separate connections from CPU -> PCIe bus 0 address space, then BARs/windows connecting PCIe bus 0 address space to individual root port/subordinate bus address space), but you're attempting to treat all 10 ports as a single PCIe domain so that you don't have to dedicate separate physical CPU address space to each port, which you would have to do if they were actually treated as separate domains.
Quoted 10 lines
Quoted 8 lines
The thing here is that when the PCIe core writes to a root port BAR window to configure/enable it the first time, you'll need to capture that transaction and dynamically allocate a window and program it in a way equivalent to what the BAR register write would have achieved on standard HW. Later, the window might need resizing, or even to be completely disabled, if the PCIe core were to change the standard BAR register. Dynamically allocating a window when the BAR is written seems a little heavy-weight.
Why?
Well, it's just a bunch more code; much more than a simple writel().
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 7 lines · Stephen Warren
Quoted 6 lines
We have 20 windows on Armada XP if I remember correctly, and they are not only used for PCIe, but also to map the BootROM (needed to boot secondary CPUs), to map SPI flashes or NOR flashes, for example. So they are really shared between many uses. In terms of PCIe, there are only two types of windows: I/O and Memory, there is no notion of Prefetchable Memory window as far as I could see.
Quoted 6 lines
In Tegra, we end up having separate MMIO vs. Prefetchable MMIO chunks of our overall PCIe aperture. However, the HW setup appears the same for both of those. I'm not sure if it's a bug in the driver, or if it's just to separate the two address spaces so that the page tables can be configured for those two regions with large rather than small granularity. I need to go investigate that.
The only purpose of prefetchable space is for legacy PCI. When a P2P bridge targets legacy PCI it has different behavior for its prefetchable memory window compared to the non-prefetchable memory window.
IIRC (though it has been a long time since I looked really close at this) PCI-X and PCI-E did away with this special bridge behaviour but kept the prefetchable memory space for compatibility.
These days it is typically used to mark cachable memory on an end device.
Quoted 1 line
From a SOC perspective, there is no need to treat MMIO and prefetch
areas any differently. ARM's per-page cachability flags can be used to deal with the differing caching requirements.
However, the bus tree downstream of each root port will require the prefetch window to be contiguous. On Marvell, today, this means you need to burn two mbus windows to get this. If the Linux pci core could allocate the prefetch space for each root port bridge contiguously with the mmio space for the same root port then this could be reduced to one window covering both spaces for the port.
Quoted 5 lines
So there are 10 PCIe interfaces (root ports). That's on the SoC itself right. Are all 10 (or a large number of them) actually used at once on any given board design? I suppose this must be the case, or Marvell wouldn't have wasted the silicon space on 10 root ports... Still, that's a rather large number of ports!
Agreed.. I have no idea what the target is for this..
Quoted 5 lines
I think the only difference on the Marvell HW is:
* The overall total size of the physical address space is dynamic rather than fixed, because it's programmed through windows rather than hard-coded into HW.
Is it hard coded on tegra? I thought there was a register set that was used to set the overall PCI-E MMIO window location and size. I know even on x86 the PCI window is set via register, though that typically isn't disclosed except to bios writers..
Jason
PATCH v2 19/27 pci: PCIe driver for Marvell Armada 370/XP systems
Quoted 1 line · Jason Gunthorpe
On Fri, Feb 01, 2013 at 10:57:20AM -0700, Stephen Warren wrote:
...
Quoted 10 lines
Quoted 5 lines
I think the only difference on the Marvell HW is:
* The overall total size of the physical address space is dynamic rather than fixed, because it's programmed through windows rather than hard-coded into HW.
Is it hard coded on tegra? I thought there was a register set that was used to set the overall PCI-E MMIO window location and size. I know even on x86 the PCI window is set via register, though that typically isn't disclosed except to bios writers..
There is a fixed (in HW) 1 GiB physical address window dedicated to PCIe. That window is divided between host controller registers, PCIe root port registers (since our root ports don't respond to configuration transactions), and regular PCIe accesses; config/MMIO/IO. There are registers in the host controller that configure the division of this space into config/MMIO/IO, so that can be dynamic. The DT bindings for the driver Thierry proposed hard-code those divisions in DT.
PATCH v2 20/27 arm: mvebu: PCIe support is now available on mvebu
In reply to Thomas PetazzoniNow that the PCIe driver for mvebu has been integrated and all its relevant dependencies, we can mark the ARCH_MVEBU platform has MIGHT_HAVE_PCI, which allows to select the PCI bus support if needed.
--- arch/arm/mach-mvebu/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/arm/mach-mvebu/Kconfig +2 -0
@@ -13,6 +13,8 @@ config ARCH_MVEBU select MVEBU_CLK_CORE select MVEBU_CLK_CPU select MVEBU_CLK_GATING + select MIGHT_HAVE_PCI + select PCI_QUIRKS if PCI if ARCH_MVEBU
Signature · 1 line
1.7.9.5
PATCH v2 21/27 arm: mvebu: add PCIe Device Tree informations for Armada 370
In reply to Thomas PetazzoniThe Armada 370 SoC has two 1x PCIe 2.0 interfaces, so we add the necessary Device Tree informations to make these interfaces availabel.
--- arch/arm/boot/dts/armada-370.dtsi | 42 +++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
arch/arm/boot/dts/armada-370.dtsi +42 -0
@@ -132,5 +132,47 @@ dmacap,memset; }; }; + + pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00001000 0 0xd0080000 0xd0080000 0 0x00002000 /* port 1.0 registers */ + 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>; + interrupt-map = <0x0800 0 0 1 &mpic 58 /* port 0.0 */ + 0x1000 0 0 1 &mpic 62>; /* port 1.0 */ + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 1,0 { + device_type = "pciex"; + reg = <0x1000 0 0xd0080000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <1>; + marvell,pcie-lane = <0>; + clocks = <&gateclk 9>; + status = "disabled"; + }; + }; }; };
Signature · 1 line
1.7.9.5
PATCH v2 22/27 arm: mvebu: add PCIe Device Tree informations for Armada XP
In reply to Thomas PetazzoniThe Armada XP SoCs have multiple PCIe interfaces. The MV78230 has 2 PCIe units (one 4x or quad 1x, the other 1x only), the MV78260 has 3 PCIe units (two 4x or quad 1x and one 4x/1x), the MV78460 has 4 PCIe units (two 4x or quad 1x and two 4x/1x). We therefore add the necessary Device Tree informations to make those PCIe interfaces usable.
--- arch/arm/boot/dts/armada-xp-mv78230.dtsi | 92 ++++++++++++++++++ arch/arm/boot/dts/armada-xp-mv78260.dtsi | 105 +++++++++++++++++++++ arch/arm/boot/dts/armada-xp-mv78460.dtsi | 150 ++++++++++++++++++++++++++++++ 3 files changed, 347 insertions(+)
arch/arm/boot/dts/armada-xp-mv78230.dtsi +92 -0
@@ -70,5 +70,97 @@ #interrupts-cells = <2>; interrupts = <87>, <88>, <89>; }; + + /* + * MV78230 has 2 PCIe units Gen2.0: One unit can be + * configured as x4 or quad x1 lanes. One unit is + * x4/x1. + */ + pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + + + bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ + + #interrupt-cells = <1>; + interrupt-map-mask = <0xf800 0 0 1>; + interrupt-map = <0x0800 0 0 1 &mpic 58 1 + 0x1000 0 0 1 &mpic 59 1 + 0x1800 0 0 1 &mpic 60 1 + 0x2000 0 0 1 &mpic 61 1 + 0x4800 0 0 1 &mpic 99 1>; + + pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 5>; + status = "disabled"; + }; + + pcie at 0,1 { + device_type = "pciex"; + reg = <0x1000 0 0xd0044000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <1>; + interrupts = <1>; + clocks = <&gateclk 6>; + status = "disabled"; + }; + + pcie at 0,2 { + device_type = "pciex"; + reg = <0x1800 0 0xd0048000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <2>; + interrupts = <1>; + clocks = <&gateclk 7>; + status = "disabled"; + }; + + pcie at 0,3 { + device_type = "pciex"; + reg = <0x2000 0 0xd004C000 0 0xC000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <3>; + interrupts = <1>; + clocks = <&gateclk 8>; + status = "disabled"; + }; + + pcie at 2,0 { + device_type = "pciex"; + reg = <0x4800 0 0xd0042000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <2>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 26>; + status = "disabled"; + }; + }; }; };
arch/arm/boot/dts/armada-xp-mv78260.dtsi +105 -0
110 lines not on this page · raw message
arch/arm/boot/dts/armada-xp-mv78460.dtsi +150 -0
155 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH v2 22/27 arm: mvebu: add PCIe Device Tree informations for Armada XP
Quoted 14 lines · Thomas Petazzoni
arch/arm/boot/dts/armada-xp-mv78230.dtsi +14
-0+ + /* + * MV78230 has 2 PCIe units Gen2.0: One unit can be + * configured as x4 or quad x1 lanes. One unit is + * x4/x1. + */ + pcie-controller { + compatible = "marvell,armada-370-xp-pcie"; + status = "disabled"; + + #address-cells = <3>; + #size-cells = <2>; + +
arch/arm/boot/dts/armada-xp-mv78230.dtsi line 96
Quoted 10 lines
arch/arm/boot/dts/armada-xp-mv78230.dtsi +10
-0+ bus-range = <0x00 0xff>; + + ranges = <0x00000800 0 0xd0040000 0xd0040000 0 0x00002000 /* port 0.0 registers */ + 0x00004800 0 0xd0042000 0xd0042000 0 0x00002000 /* port 2.0 registers */ + 0x00001000 0 0xd0044000 0xd0044000 0 0x00002000 /* port 0.1 registers */ + 0x00001800 0 0xd0048000 0xd0048000 0 0x00002000 /* port 0.2 registers */ + 0x00002000 0 0xd004C000 0xd004C000 0 0x00002000 /* port 0.3 registers */ + 0x81000000 0 0 0xc0000000 0 0x00010000 /* downstream I/O */ + 0x82000000 0 0 0xc1000000 0 0x08000000>; /* non-prefetchable memory */ +
I've been thinking some more about this, and I wonder if it would make more sense to describe the address remapping correctly as a node on top of the pcie-controller node.
This would mean that rather than putting the mapped physical address (0xc0000000, 0xc1000000, ...) in here, you would actually have 64-bit address as the destination as well, in whatever format the address map hardware uses, I assume using a numbered 32 bit address space for each object that can be remapped.
This would also let you do the PCI memory address assignment for each port separately, starting at bus address 0, followed by finding a location in the CPU address space and passing the start as the sys->mem_offset argument to pci_add_resource_offset.
arch/arm/boot/dts/armada-xp-mv78230.dtsi line 115
Quoted 11 lines
arch/arm/boot/dts/armada-xp-mv78230.dtsi +11
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0 0x2000>; + #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 5>; + status = "disabled"; + };
I think you are missing a "ranges" property here, at least an empty one, which is required by the standard but not currently enforced in the code.
Arnd
PATCH v2 22/27 arm: mvebu: add PCIe Device Tree informations for Armada XP
Dear Arnd Bergmann,
Quoted 10 lines · Arnd Bergmann
I've been thinking some more about this, and I wonder if it would make more sense to describe the address remapping correctly as a node on top of the pcie-controller node.
This would mean that rather than putting the mapped physical address (
0xc0000000,0xc1000000, ...) in here, you would actually have 64-bit address as the destination as well, in whatever format the address map hardware uses, I assume using a numbered 32 bit address space for each object that can be remapped.
Quoted 5 lines
This would also let you do the PCI memory address assignment for each port separately, starting at bus address 0, followed by finding a location in the CPU address space and passing the start as the
sys->mem_offsetargument topci_add_resource_offset.
Hum, good you give a skeleton example, because I'm not sure to understand your suggestion.
Quoted 14 lines
arch/arm/boot/dts/armada-xp-mv78230.dtsi line 115
Quoted 13 lines
arch/arm/boot/dts/armada-xp-mv78230.dtsi +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0
0x2000>;arch/arm/boot/dts/armada-xp-mv78230.dtsi +8
-0+ #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 5>; + status = "disabled"; + };
Quoted 3 lines
I think you are missing a "ranges" property here, at least an empty one, which is required by the standard but not currently enforced in the code.
Is it really wise to have DT properties that are not used by anything, and therefore have a very high chance of either being incorrect, or becoming incorrect?
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 22/27 arm: mvebu: add PCIe Device Tree informations for Armada XP
Quoted 14 lines · Thomas Petazzoni
Dear Arnd Bergmann,
Quoted 10 lines · Arnd Bergmann
I've been thinking some more about this, and I wonder if it would make more sense to describe the address remapping correctly as a node on top of the pcie-controller node.
This would mean that rather than putting the mapped physical address (
0xc0000000,0xc1000000, ...) in here, you would actually have 64-bit address as the destination as well, in whatever format the address map hardware uses, I assume using a numbered 32 bit address space for each object that can be remapped.
Quoted 8 lines
Quoted 5 lines
This would also let you do the PCI memory address assignment for each port separately, starting at bus address 0, followed by finding a location in the CPU address space and passing the start as the
sys->mem_offsetargument topci_add_resource_offset.Hum, good you give a skeleton example, because I'm not sure to understand your suggestion.
I mean (roughly, since I don't know how that hardware defines it)
/ {
#address-cells = <1>;
#size-cells = <1>;
memory at 0 {
/* this node can not get remapped */
reg = <0x0 0x40000000>;
};
address-map {
/* this device translates 64 bit MMIO bus addresses into 32 bit CPU addresses */
compatible = "marvell,armada-addr-decoding-controller";
reg = <0xd0020000 0x258>;
#addres-cells = <2>;
#address-cells = <1>;
/* each remapped window has one entry here */
ranges = <0xa 0 0xc0000000 0x10000>, /* map window a to 0xc0000000 */
<0xb 0 0xc1000000 0x08000000>, /* map window b to 0xc1000000 */
<...>; /* more windows */
pciex {
#addres-cells = <3>;
#size-cells = <2>;
ranges = <0x81000000 0 0 0xa 0 0 0x00010000, /* I/O is window a */
0x82000000 0 0 0xb 0 0 0x08000000>; /* non-prefetchable memory */
...
};
something-else {
...
reg = <0xc 0 0x10000>; /* window c */
};
};
};Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
arch/arm/boot/dts/armada-xp-mv78230.dtsi +3
-0+ pcie at 0,0 { + device_type = "pciex"; + reg = <0x0800 0 0xd0040000 0
0x2000>;arch/arm/boot/dts/armada-xp-mv78230.dtsi +8
-0+ #address-cells = <3>; + #size-cells = <2>; + marvell,pcie-port = <0>; + marvell,pcie-lane = <0>; + interrupts = <1>; + clocks = <&gateclk 5>; + status = "disabled"; + };
Quoted 7 lines
Quoted 3 lines
I think you are missing a "ranges" property here, at least an empty one, which is required by the standard but not currently enforced in the code.
Is it really wise to have DT properties that are not used by anything, and therefore have a very high chance of either being incorrect, or becoming incorrect?
In this case, definitely. It's mandated by an IEEE standard and the only reason why we let some mistakes slip here is that some ancient PowerMac systems get it wrong in their firmware. There are lots of cases where the difference between an empty "ranges" property and an absent one is very significant and I would really prefer to change the kernel to be standard compliant here.
Arnd