pci and pcie device-tree binding - range No cells
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (xpedite5370.dts, sequoia.dts, etc) but also 6 cells format too (mpc832x_mds.dts)
Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_Bridge
And also in documentation in the linux Documentation/devicetree/bindings/pci/83xx-512x-pci.txt
Both format uses:
#size-cells = <2>;
#address-cells = <3>;
What is valid format?
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 12 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_Bridge
Quoted 8 lines
And also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Rob
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;
Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 {
#address-cells = <1>;
#size-cells = <1>;
axi_pcie_0: axi-pcie@50000000 {
#address-cells = <3>;
#size-cells = <2>;
compatible = "xlnx,axi-pcie-1.05.a";
ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >;
...
}
}What I am wondering is pci_process_bridge_OF_ranges() at arch/powerpc/kernel/pci-common.c where there are used some hardcoded values which should be probably loaded from device-tree.
For example:
683 int np = pna + 5;
...
702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);
Unfortunately we have copied it to microblaze.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 50 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.
Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);
These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Quoted 3 lines
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like drivers/pci/pci-of.c would be great.
Rob
Quoted 9 lines
Thanks,
Michal
Quoted 58 lines · Rob Herring
Quoted 50 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);
Quoted 2 lines
These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Sorry for my pci ignorance (have never got hw for mb/zynq)
I just want to get better overview how we should we our drivers to be compatible.
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
Quoted 8 lines
Quoted 1 line
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like
drivers/pci/pci-of.cwould be great.
We have done pcie working on zynq and the same pcie host is working on Microblaze too. There are just small differences. That's why I have sent another email ("Sharing PCIE driver between Microblaze and Arm zynq") to find out proper location.
PCI: Microblaze shares almost the same code with powerpc that's why I am definitely open to move this code out of architecture.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 2 lines
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
I certainly believe that all PCIe (not PCI) transfers are nominally multiples of 64bit data.
There are (effectively) 8 byte-lane enables to allow partial word transfers (I'm not sure whether disjoint subwords are valid).
Some PCIe slave->32bit bus bridges always do two slave cycles. Even though, for a 32bit word transfer, one of them ends up having no byte enables asserted.
David
Quoted 5 lines · David Laight
Quoted 2 lines
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
I certainly believe that all PCIe (not PCI) transfers are nominally multiples of 64bit data.
And PCI? That powerpc/pci-common code was designed for PCI and microblaze also used it for PCI.
CC: Thomas: I think it will be interesting to see this discussion because you are using size-cell/address-cells equal 1.
http://www.spinics.net/lists/arm-kernel/msg211839.html
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Dear Michal Simek,
Quoted 3 lines · Michal Simek
CC: Thomas: I think it will be interesting to see this discussion because you are using size-cell/address-cells equal 1.
http://www.spinics.net/lists/arm-kernel/msg211839.html
Thanks for Cc'ing me.
The thing is that on Marvell SoCs, we don't need to describe statically in the Device Tree the translation between CPU addresses and PCI device addresses, because those translations are set up dynamically at run time through address decoding windows.
Marvell SoCs have up to 20 configurable address windows, which allow you, at run time, to say "I would like the range from physical address 0xYYYYYYYY to 0xZZZZZZZZ to correspond to the PCIe device in port 1, lane 2, or to the NAND, or to this or that device". Therefore, in the PCIe driver I proposed for the Armada 370/XP SoCs [1], there is no need to encode all those ranges statically in the DT.
The only "ranges" property I'm using is to allow the DT sub-nodes describing each PCIe port/lane to access the CPU registers that allow to see if the PCIe link is up or down, access the PCI configuration space and so on. So all ranges in my "ranges" property correspond to normal CPU registers, like the one you would put in the "reg" property for any device. The fact that those devices are PCIe is really orthogonal here.
Of course, I have no idea if I'm doing a correct usage of the DT, but I certainly don't need those 6 values ranges with bits to say if it's I/O space, memory space, bus number, device number and so on. The physical addresses at which I'm setting up my address decoding windows are decided dynamically at runtime, depending on the number of PCIe devices that are found in the different PCIe slots (for now, those windows have a statically defined size, but I'd ideally would like to size them to match exactly the size of the PCIe device memory, in order to avoid wasting physical address space, but I haven't found how to get the size needed for each PCIe device during the ARM pcibios initialization sequence).
For those willing to have a look at the PCIe patch set for Armada 370/XP, see:
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/136455.html.
The core of the PCIe driver and its DT binding documentation is at
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/136711.html.
Thanks a lot for your comments,
Thomas
[1] that I hope to extend to cover previous Marvell SoCs as well, they work basically the same way
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 15 lines · Thomas Petazzoni
Dear Michal Simek,
Quoted 3 lines · Michal Simek
CC: Thomas: I think it will be interesting to see this discussion because you are using size-cell/address-cells equal 1.
http://www.spinics.net/lists/arm-kernel/msg211839.htmlThanks for Cc'ing me.
The thing is that on Marvell SoCs, we don't need to describe statically in the Device Tree the translation between CPU addresses and PCI device addresses, because those translations are set up dynamically at run time through address decoding windows.
Quoted 6 lines
Marvell SoCs have up to 20 configurable address windows, which allow you, at run time, to say "I would like the range from physical address 0xYYYYYYYY to 0xZZZZZZZZ to correspond to the PCIe device in port 1, lane 2, or to the NAND, or to this or that device". Therefore, in the PCIe driver I proposed for the Armada 370/XP SoCs [1], there is no need to encode all those ranges statically in the DT.
That's not a unique feature. I'm not sure if any powerpc systems do that though.
Quoted 7 lines
The only "ranges" property I'm using is to allow the DT sub-nodes describing each PCIe port/lane to access the CPU registers that allow to see if the PCIe link is up or down, access the PCI configuration space and so on. So all ranges in my "ranges" property correspond to normal CPU registers, like the one you would put in the "reg" property for any device. The fact that those devices are PCIe is really orthogonal here.
That doesn't really sound right.
Quoted 11 lines
Of course, I have no idea if I'm doing a correct usage of the DT, but I certainly don't need those 6 values ranges with bits to say if it's I/O space, memory space, bus number, device number and so on. The physical addresses at which I'm setting up my address decoding windows are decided dynamically at runtime, depending on the number of PCIe devices that are found in the different PCIe slots (for now, those windows have a statically defined size, but I'd ideally would like to size them to match exactly the size of the PCIe device memory, in order to avoid wasting physical address space, but I haven't found how to get the size needed for each PCIe device during the ARM pcibios initialization sequence).
I don't think deviating from the normal binding is the right approach. Perhaps the host driver should fill in the ranges property with the addresses it uses. Then any child devices will get the right address translation.
Also, while the h/w may support practically any config, there are practical constraints of what Linux will use like there's no reason to support more than 64K i/o space. PCI memory addresses generally start at 0x100000. You probably don't need more than 1 memory window per root complex (although prefetchable memory may also be needed).
You could let the DT settings drive the address window configuration.
Rob
Quoted 13 lines
For those willing to have a look at the PCIe patch set for Armada 370/XP, see:
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/136455.html.
The core of the PCIe driver and its DT binding documentation is at
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/136711.html.Thanks a lot for your comments,
Thomas
[1] that I hope to extend to cover previous Marvell SoCs as well, they work basically the same way
Dear Rob Herring,
Quoted 10 lines · Rob Herring
Quoted 7 lines
Marvell SoCs have up to 20 configurable address windows, which allow you, at run time, to say "I would like the range from physical address 0xYYYYYYYY to 0xZZZZZZZZ to correspond to the PCIe device in port 1, lane 2, or to the NAND, or to this or that device". Therefore, in the PCIe driver I proposed for the Armada 370/XP SoCs [1], there is no need to encode all those ranges statically in the DT.
That's not a unique feature. I'm not sure if any powerpc systems do that though.
Yes, probably not an unique feature.
Quoted 9 lines
Quoted 7 lines
The only "ranges" property I'm using is to allow the DT sub-nodes describing each PCIe port/lane to access the CPU registers that allow to see if the PCIe link is up or down, access the PCI configuration space and so on. So all ranges in my "ranges" property correspond to normal CPU registers, like the one you would put in the "reg" property for any device. The fact that those devices are PCIe is really orthogonal here.
That doesn't really sound right.
Very likely, but I still don't get what is "the right way".
Quoted 4 lines
I don't think deviating from the normal binding is the right approach. Perhaps the host driver should fill in the ranges property with the addresses it uses. Then any child devices will get the right address translation.
I don't really understand what you mean here. If you look at the host driver code (arch/arm/mach-mvebu/pcie.c), for each PCIe interface is simply does:
* Create an address decoding window for the memory BAR
* Create an address decoding window for the I/O BAR
* Associate the memory BAR window address and the I/O bar window address with the PCIe interface
And that's it. See
https://github.com/MISL-EBU-System-SW/mainline-public/blob/marvell-pcie-v1/arch/arm/mach-mvebu/pcie.c#L107.
So this driver is both "deciding" of the physical addresses for each PCIe interface, and "associating" them with the PCIe interfaces. How is it useful to feed some addresses back into the Device Tree?
Quoted 5 lines
Also, while the h/w may support practically any config, there are practical constraints of what Linux will use like there's no reason to support more than 64K i/o space. PCI memory addresses generally start at
0x100000. You probably don't need more than 1 memory window per root complex (although prefetchable memory may also be needed).
I allocate one 64K I/O window and one memory window per PCIe interface whose link is up (i.e a PCIe device is connected).
Quoted 1 line
You could let the DT settings drive the address window configuration.
No, because I don't want to have absolute addresses for the windows: I have 10 PCIe interfaces, but often, only a few of them are used. So I don't want in the Device Tree to over-allocate hundreds of MB of physical address space if it's not useful.
PCIe is dynamic, address window configuration is dynamic. And we should hardcode all this configuration statically in the DT? Doesn't seem like the right solution.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 25 lines · Thomas Petazzoni
Dear Rob Herring,
Quoted 10 lines · Rob Herring
Quoted 7 lines
Marvell SoCs have up to 20 configurable address windows, which allow you, at run time, to say "I would like the range from physical address 0xYYYYYYYY to 0xZZZZZZZZ to correspond to the PCIe device in port 1, lane 2, or to the NAND, or to this or that device". Therefore, in the PCIe driver I proposed for the Armada 370/XP SoCs [1], there is no need to encode all those ranges statically in the DT.
That's not a unique feature. I'm not sure if any powerpc systems do that though.
Yes, probably not an unique feature.
Quoted 8 lines
Quoted 7 lines
The only "ranges" property I'm using is to allow the DT sub-nodes describing each PCIe port/lane to access the CPU registers that allow to see if the PCIe link is up or down, access the PCI configuration space and so on. So all ranges in my "ranges" property correspond to normal CPU registers, like the one you would put in the "reg" property for any device. The fact that those devices are PCIe is really orthogonal here.
Quoted 3 lines
Quoted 1 line
That doesn't really sound right.
Very likely, but I still don't get what is "the right way".
Hi Thomas,
I just went and looked at your binding. Here's the snippet I found interesting:
pcie-controller {
+ compatible = "marvell,armada-370-xp-pcie";
+ status = "disabled";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges = <0 0xd0040000 0x2000 /* port0x1_port0 */
+ 0x2000 0xd0042000 0x2000 /* port2x1_port0 */ + 0x4000 0xd0044000 0x2000 /* port0x1_port1 */ + 0x8000 0xd0048000 0x2000 /* port0x1_port2 */ + 0xC000 0xd004C000 0x2000 /* port0x1_port3 */ + 0x10000 0xd0080000 0x2000 /* port1x1_port0 */ + 0x12000 0xd0082000 0x2000 /* port3x1_port0 */ + 0x14000 0xd0084000 0x2000 /* port1x1_port1 */ + 0x18000 0xd0088000 0x2000 /* port1x1_port2 */ + 0x1C000 0xd008C000 0x2000 /* port1x1_port3 */>;
+
+ pcie0.0 at 0xd0040000 {
+ reg = <0x0 0x2000>;
+ interrupts = <58>;
+ clocks = <&gateclk 5>;
+ marvell,pcie-port = <0>;
+ marvell,pcie-lane = <0>;
+ status = "disabled";
+ };
+
+ pcie0.1 at 0xd0044000 {
+ reg = <0x4000 0x2000>;
+ interrupts = <59>;
+ clocks = <&gateclk 5>;
+ marvell,pcie-port = <0>;
+ marvell,pcie-lane = <1>;
+ status = "disabled";
+ };
[... rest trimmed for berevity]
You're right, if you're doing dynamic allocation of windows, then you really don't need to have a ranges property. However, the way the PCI node is set up definitely looks incorrect.
PCI already has a very specific binding for pci host controller nodes. First, #address-cells=<3>; #size-cells=<2>; and device_type="pcie" must be there. You don't want to break this. You can find details on the pci and pci-express binding here:
http://www.openfirmware.org/1275/bindings/pci/pci2_1.pdf
http://www.openfirmware.org/1275/bindings/pci/pci-express.txt
For the child nodes, PCI is a discoverable bus, so normally I wouldn't expect to see child nodes at all when using a dtb. The only time nodes should be populated is when a device has non-discoverable charactersitics. In your example above you do have some additional data, but I don't know enough about pci-express to say how best to encode them or whether they are needed at all. Ben might have some thoughts on this.
When the PCI child nodes are present, it is important to stick with the established PCI addressing scheme which uses 3 cells for addressing. The first entry in the reg property must represent the configuration space so that DT nodes can be matched up with discovered devices. There is no requirement to include mappings for the memory and io regions if the host controller can assign them dynamically.
I don't think you should need a ranges property at all for what you're doing. Access to config space is generally managed by the PCI host controller drivers and subsystem, and PCI device drivers don't typically use of_ calls directly.
g.
Quoted 43 lines · Thomas Petazzoni
Dear Rob Herring,
Quoted 10 lines · Rob Herring
Quoted 7 lines
Marvell SoCs have up to 20 configurable address windows, which allow you, at run time, to say "I would like the range from physical address 0xYYYYYYYY to 0xZZZZZZZZ to correspond to the PCIe device in port 1, lane 2, or to the NAND, or to this or that device". Therefore, in the PCIe driver I proposed for the Armada 370/XP SoCs [1], there is no need to encode all those ranges statically in the DT.
That's not a unique feature. I'm not sure if any powerpc systems do that though.
Yes, probably not an unique feature.
Quoted 9 lines
Quoted 7 lines
The only "ranges" property I'm using is to allow the DT sub-nodes describing each PCIe port/lane to access the CPU registers that allow to see if the PCIe link is up or down, access the PCI configuration space and so on. So all ranges in my "ranges" property correspond to normal CPU registers, like the one you would put in the "reg" property for any device. The fact that those devices are PCIe is really orthogonal here.
That doesn't really sound right.
Very likely, but I still don't get what is "the right way".
Quoted 4 lines
I don't think deviating from the normal binding is the right approach. Perhaps the host driver should fill in the ranges property with the addresses it uses. Then any child devices will get the right address translation.
I don't really understand what you mean here. If you look at the host driver code (
arch/arm/mach-mvebu/pcie.c), for each PCIe interface is simply does:* Create an address decoding window for the memory BAR
* Create an address decoding window for the I/O BAR
* Associate the memory BAR window address and the I/O bar window address with the PCIe interface
Quoted 6 lines
And that's it. See
https://github.com/MISL-EBU-System-SW/mainline-public/blob/marvell-pcie-v1/arch/arm/mach-mvebu/pcie.c#L107.So this driver is both "deciding" of the physical addresses for each PCIe interface, and "associating" them with the PCIe interfaces. How is it useful to feed some addresses back into the Device Tree?
I'm not completely sure for PCI, but the ranges is necessary to translate addresses of child nodes.
If you don't need ranges then you could omit it. If you need ranges, then you should follow the PCI binding whether it is put in the DTS or you dynamically fill it in. This could be filled in by the bootloader as well if you have PCI devices you need to boot from.
Quoted 9 lines
Quoted 5 lines
Also, while the h/w may support practically any config, there are practical constraints of what Linux will use like there's no reason to support more than 64K i/o space. PCI memory addresses generally start at
0x100000. You probably don't need more than 1 memory window per root complex (although prefetchable memory may also be needed).I allocate one 64K I/O window and one memory window per PCIe interface whose link is up (i.e a PCIe device is connected).
Quoted 6 lines
Quoted 1 line
You could let the DT settings drive the address window configuration.
No, because I don't want to have absolute addresses for the windows: I have 10 PCIe interfaces, but often, only a few of them are used. So I don't want in the Device Tree to over-allocate hundreds of MB of physical address space if it's not useful.
How many you have is probably board dependent and not probe-able, right? So you would at least know the subset of root complexes that you are using. I know you want to find the size of all the cards up front and size windows based on that, but I don't think that is going to be possible.
Quoted 4 lines
PCIe is dynamic, address window configuration is dynamic. And we should hardcode all this configuration statically in the DT? Doesn't seem like the right solution.
I'm just throwing out ideas. There are many cases of flexibility in h/w designs which are never used. H/w is often designed in a vacuum without s/w input. Not saying that is the case here, but you do have to consider that.
Rob
Quoted 5 lines
Best regards,
Thomas
Quoted 64 lines · Michal Simek
Quoted 60 lines · Rob Herring
Quoted 52 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 7 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells
are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);Quoted 2 lines
These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Quoted 6 lines
Sorry for my pci ignorance (have never got hw for mb/zynq) I just want to get better overview how we should we our drivers to be compatible.
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
That's what the DT documentation says.
BTW, you can use 2 cells even if you only have a 32-bit bus.
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like
drivers/pci/pci-of.cwould be great.
Quoted 5 lines
We have done pcie working on zynq and the same pcie host is working on Microblaze too.
There are just small differences. That's why I have sent another email ("Sharing PCIE driver between Microblaze and Arm zynq") to find out proper location.
Yes, I've followed the thread. There are lots of areas for consolidation with PCIe hosts both across architectures and within ARM. There are multiple people using DW PCIe IP although not upstream yet that I'm aware of.
Rob
Quoted 10 lines
PCI: Microblaze shares almost the same code with powerpc that's why I am definitely open
to move this code out of architecture.Thanks,
Michal
Quoted 69 lines · Rob Herring
Quoted 68 lines · Michal Simek
Quoted 60 lines · Rob Herring
Quoted 52 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 7 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells
are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);Quoted 2 lines
These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Sorry for my pci ignorance (have never got hw for mb/zynq) I just want to get better overview how we should we our drivers to be compatible.
Quoted 6 lines
Quoted 2 lines
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
That's what the DT documentation says.
BTW, you can use 2 cells even if you only have a 32-bit bus.
ok - no problem with that. Just want to be sure about it.
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like
drivers/pci/pci-of.cwould be great.Quoted 5 lines
We have done pcie working on zynq and the same pcie host is working on Microblaze too.
There are just small differences. That's why I have sent another email ("Sharing PCIE driver between Microblaze and Arm zynq") to find out proper location.
Quoted 4 lines
Yes, I've followed the thread. There are lots of areas for consolidation with PCIe hosts both across architectures and within ARM. There are multiple people using DW PCIe IP although not upstream yet that I'm aware of.
Your suggestions are really appreciate.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 59 lines · Michal Simek
Quoted 58 lines · Rob Herring
Quoted 56 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);
Quoted 8 lines
Quoted 2 lines
These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Sorry for my pci ignorance (have never got hw for mb/zynq)
I just want to get better overview how we should we our drivers to be compatible.Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
Yes, PCIe addressing is always 64 bits wide. Even on 32bit PCI systems we use 64 bit PCI addressing in the device tree.
g.
Quoted 9 lines · Grant Likely
Quoted 2 lines
Sorry for my pci ignorance (have never got hw for mb/zynq) I just want to get better overview how we should we our drivers to
be compatible.
Quoted 3 lines
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
Yes, PCIe addressing is always 64 bits wide. Even on 32bit PCI systems we use 64 bit PCI addressing in the device tree.
Right. The size & format of an address cell for PCI is specified in the OF PCI bindings and we follow that binding. It's always 3 cells.
Cheers,
Ben.
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 4 lines · Grant Likely
Quoted 2 lines
Sorry for my pci ignorance (have never got hw for mb/zynq) I just want to get better overview how we should we our drivers to
be compatible.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Does it mean that pci is supposed be always 64 bit wide? And there is no option to have just 32bit values.
Yes, PCIe addressing is always 64 bits wide. Even on 32bit PCI systems we use 64 bit PCI addressing in the device tree.
Right. The size & format of an address cell for PCI is specified in the OF PCI bindings and we follow that binding. It's always 3 cells.
.. and the reason why it must be 3 cells, even if the host PCI bus only supports 32-bit addressing, is because a plug-in PCI card has no way of knowing what the host supports.
Quoted 10 lines
Cheers,
Ben._______________________________________________
devicetree-discuss mailing list devicetree-discuss-uLR06cmDAlY/[email protected]
https://lists.ozlabs.org/listinfo/devicetree-discuss
Quoted 63 lines · Rob Herring
Quoted 50 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Quoted 2 lines
Quoted 8 lines
Quoted 1 line
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like
drivers/pci/pci-of.cwould be great.
A lot of it should be common. The microblaze code is a copy of the powerpc version. I'll strongly nack any attempt to add a third! :-)
drivers/pci/pci-of.c would be good. I'd also accept drivers/of/pci.c which might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.
g.
Quoted 73 lines · Grant Likely
Quoted 63 lines · Rob Herring
Quoted 50 lines · Michal Simek
Quoted 20 lines · Rob Herring
Quoted 19 lines · Michal Simek
Hi Grant and others,
I have a question regarding number of cells in ranges property for pci and pcie nodes.
Linux pci/pcie powerpc DTSes contain 7 cells (
xpedite5370.dts,sequoia.dts, etc)
but also 6 cells format too (mpc832x_mds.dts)Here is shown 6 cells ranges format and describe
http://devicetree.org/Device_Tree_Usage#PCI_Host_BridgeAnd also in documentation in the linux
Documentation/devicetree/bindings/pci/83xx-512x-pci.txtBoth format uses:
#size-cells = <2>;
#address-cells = <3>;Quoted 6 lines
Quoted 1 line
What is valid format?
Both. 7 cells are valid when the host (parent) bus is 64-bit and 6 cells are valid when the host bus is 32-bit. The ranges property is <<child address> <parent address> <size>>. The parent address #address-cells is taken from the parent node.
Ok. Got it.
Here is what we use on zynq and microblaze - both 32bit which should be fine.
ps7_axi_interconnect_0: axi@0 { #address-cells = <1>; #size-cells = <1>; axi_pcie_0: axi-pcie@50000000 { #address-cells = <3>; #size-cells = <2>; compatible = "xlnx,axi-pcie-1.05.a"; ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >; ... } }What I am wondering is
pci_process_bridge_OF_ranges()atarch/powerpc/kernel/pci-common.c
where there are used some hardcoded values which should be probably loaded from device-tree.Quoted 6 lines
For example:
683 int np = pna + 5;
...702 pci_addr = of_read_number(ranges + 1, 2); 703 cpu_addr = of_translate_address(dev, ranges + 3); 704 size = of_read_number(ranges + pna + 3, 2);These would always be correct whether you have 6 or 7 cells. pna is the parent bus address cells size. The pci address is fixed at 3 cells.
Quoted 2 lines
Quoted 8 lines
Quoted 1 line
Unfortunately we have copied it to microblaze.
I look at the PCI DT code in powerpc and see a whole bunch of code that seems like it should be common. The different per arch pci structs complicates that. No one has really gotten to looking at PCI DT on ARM yet except you and Thierry for Tegra. We definitely don't want to create a 3rd copy. Starting the process of moving it to something like
drivers/pci/pci-of.cwould be great.
Quoted 2 lines
A lot of it should be common. The microblaze code is a copy of the powerpc version. I'll strongly nack any attempt to add a third! :-)
Yes it. There are some things which we had fixed because that powerpc port is big endian only and we support PCIe on little endian too. But changes are really cosmetic.
Quoted 4 lines
drivers/pci/pci-of.cwould be good. I'd also acceptdrivers/of/pci.cwhich might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.
Ben: Are you willing to move that ppc code to this location? It is probably not good idea that I should do it when I even don't have hardware available for testing (Asking someone else).
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 9 lines · Michal Simek
Quoted 4 lines · Grant Likely
drivers/pci/pci-of.cwould be good. I'd also acceptdrivers/of/pci.cwhich might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.Ben: Are you willing to move that ppc code to this location? It is probably not good idea that I should do it when I even don't have hardware available for testing (Asking someone else).
You're a clever guy, you are more than capable of crafting the patch, even if you can't test on hardware. :-)
I refactored most of the OF support code without having access to most of the affected hardware. Once I got the changes out there for review I also asked for spot testing before getting it into linux-next for even more testing.
g.
A message that is not in the archive (sent off-list or not yet received): CAPcvp5EJH-Q6wd7my+V+FUVE1=hzwMN-yOfHiukGvDmkcoRcsQ@mail.gmail.com
Quoted 11 lines · Grant Likely
Quoted 9 lines · On Wed, Dec 12, 2012 at 10:37 AM, Michal Simek…
Quoted 4 lines · Grant Likely
drivers/pci/pci-of.cwould be good. I'd also acceptdrivers/of/pci.cwhich might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.Ben: Are you willing to move that ppc code to this location? It is probably not good idea that I should do it when I even don't have hardware available for testing (Asking someone else).
Quoted 7 lines
You're a clever guy, you are more than capable of crafting the patch, even if you can't test on hardware. :-)
I refactored most of the OF support code without having access to most of the affected hardware. Once I got the changes out there for review I also asked for spot testing before getting it into linux-next for even more testing.
I've been working on a relatively architecture agnostic PCI host bridge driver and also wanted to avoid duplicating more generic DT parsing code for PCI bindings.
I've ended up with a patch which provides an iterator for returning resources based on the the typical 'ranges' binding. This has ended up living in drivers/of/address.c. I originally started out in drivers/of/pci.c and drivers/pci/pci-of.c but found there were good (and static) implementations in drivers/of/address.c which can be reused (e.g. of_bus_pci_get_flags, bus->count_cells).
I'm not just ready to post it - but can do before early next week if you can wait.
Andrew Murray
Quoted 31 lines · Andrew Murray
Quoted 11 lines · Grant Likely
Quoted 9 lines · On Wed, Dec 12, 2012 at 10:37 AM, Michal Simek…
Quoted 4 lines · Grant Likely
drivers/pci/pci-of.cwould be good. I'd also acceptdrivers/of/pci.cwhich might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.Ben: Are you willing to move that ppc code to this location? It is probably not good idea that I should do it when I even don't have hardware available for testing (Asking someone else).
Quoted 7 lines
You're a clever guy, you are more than capable of crafting the patch, even if you can't test on hardware. :-)
I refactored most of the OF support code without having access to most of the affected hardware. Once I got the changes out there for review I also asked for spot testing before getting it into linux-next for even more testing.
I've been working on a relatively architecture agnostic PCI host bridge driver and also wanted to avoid duplicating more generic DT parsing code for PCI bindings.
I've ended up with a patch which provides an iterator for returning resources based on the the typical 'ranges' binding. This has ended up living in
drivers/of/address.c. I originally started out indrivers/of/pci.canddrivers/pci/pci-of.cbut found there were good (and static) implementations indrivers/of/address.cwhich can be reused (e.g.of_bus_pci_get_flags,bus->count_cells).
Quoted 2 lines
I'm not just ready to post it - but can do before early next week if you can wait.
I already posted a similar patch[0] as part of a larger series to bring DT support to Tegra PCIe back in July. I suppose what you have must be something pretty close to that. Most of the stuff that had me occupied since then should be done soon and I was planning on resurrecting the series one of these days.
Thierry
Quoted 12 lines · Thierry Reding
Quoted 11 lines · Andrew Murray
I've been working on a relatively architecture agnostic PCI host bridge driver and also wanted to avoid duplicating more generic DT parsing code for PCI bindings.
I've ended up with a patch which provides an iterator for returning resources based on the the typical 'ranges' binding. This has ended up living in
drivers/of/address.c. I originally started out indrivers/of/pci.canddrivers/pci/pci-of.cbut found there were good (and static) implementations indrivers/of/address.cwhich can be reused (e.g.of_bus_pci_get_flags,bus->count_cells).
Quoted 8 lines
Quoted 2 lines
I'm not just ready to post it - but can do before early next week if you can wait.
I already posted a similar patch[0] as part of a larger series to bring DT support to Tegra PCIe back in July. I suppose what you have must be something pretty close to that. Most of the stuff that had me occupied since then should be done soon and I was planning on resurrecting the series one of these days.
Thanks for the reference. I've submitted my patch, it's along the lines of your existing patch.
I'm happy to take the best bits from both, drop mine, etc.
Andrew Murray
Quoted 11 lines · Grant Likely
Quoted 9 lines · Michal Simek
Quoted 4 lines · Grant Likely
drivers/pci/pci-of.cwould be good. I'd also acceptdrivers/of/pci.cwhich might actually be a good idea in the short term so that it gets appropriate supervision while being generalized before being moved into the pci directory.Ben: Are you willing to move that ppc code to this location? It is probably not good idea that I should do it when I even don't have hardware available for testing (Asking someone else).
Quoted 7 lines
You're a clever guy, you are more than capable of crafting the patch, even if you can't test on hardware. :-)
I refactored most of the OF support code without having access to most of the affected hardware. Once I got the changes out there for review I also asked for spot testing before getting it into linux-next for even more testing.
Fair enough. :-)
Good time to start to look for how to work with board farm.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian