PCI Patches update
Hi all,
I got email from Ivan Kokshaysky describing recent changes in 2.5 PCI support and realized I should document one of the outstanding "bugs" with PCI resource management under PARISC.
enjoy!
grant
---- Forwarded message from Grant Grundler <[email protected]> -----
Quoted 1 line · Ivan Kokshaysky
- windows overlapping with something else.
On parisc, this is one of the problems I've dodged.
I'm not looking for a solution right now - but if you know the obvious answer, please tell me. I just want to outline the problem.
The PCI controller (lba_pci.c) on newer PARISC machines can support two MMIO ranges. The "Distributed" range is a "fixed" size and base (1/8th of whatever the parent IO Controller (SBA) has). The second "Directed" Range can use any address otherwise not assigned to PCI devices and directed to any PCI controller under the same SBA.
PARISC lba_pci.c currently only advertises Distributed ranges and pretends Directed ranges don't exist. Things still work, but the resource accounting in /proc/iomem is inaccurate.
Since Directed Ranges have precedence in the MMIO address routing, they can overlap with Distributed ranges. Worse, The SBA can assign "duplicate" MMIO address ranges by routing "unused" address space to a Directed range under a different PCI controller. I think both are very ugly scenarios and will need alot of code to detect/handle properly.
Each Distributed Range is typically only 16MB in size. Consumers of Directed Ranges are things like graphics cards or X.25 cards which want 16MB or greater in a single BAR.
Eg /proc/iomem on a C3000 running 2.4.20 says:
f0000000-f0ffffff : LBA PCI LMMIO
f05d0000-f05d0000 : lcd_data
f05d0008-f05d0008 : lcd_cmd
f1000000-f1ffffff : LBA PCI LMMIO
f4000000-f4ffffff : LBA PCI LMMIO
f4007000-f4007fff : usb-ohci
f4008000-f40083ff : tulip
f4900000-f490007f : tulip
f4901000-f490107f : tulip
f4902000-f490207f : tulip
f4903000-f490307f : tulip
f6000000-f6ffffff : LBA PCI LMMIO
f6000000-f600ffff : tg3
fa100000-fa4fffff : stifb mmio
fb000000-fbffffff : stifb
...
In the same machine (C3000) as the above, lspci says:
...
01:04.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, medium devsel, latency 255 Bus: primary=01, secondary=02, subordinate=02, sec-latency=255 I/O behind bridge: 00003000-00003fff Memory behind bridge: f4900000-f49fffff Prefetchable memory behind bridge: 00000000f0100000-00000000f0100000 Capabilities: <available only to root> ... 03:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01) Subsystem: Hewlett-Packard Company: Unknown device 1340 Flags: bus master, 66Mhz, medium devsel, latency 255, IRQ 163 I/O ports at 28000 [size=256] Memory at f6014000 (64-bit, non-prefetchable) [size=1K] Memory at f6010000 (64-bit, non-prefetchable) [size=8K] Expansion ROM at f6020000 [disabled] [size=128K] Capabilities: <available only to root>
03:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet (rev 02)
Subsystem: Broadcom Corporation NetXtreme BCM5703 1000Base-T
Flags: bus master, 66Mhz, medium devsel, latency 248, IRQ 160
Memory at f6000000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <available only to root>
04:02.0 Display controller: Hewlett-Packard Company A4977A Visualize EG (rev 03)
Flags: 66Mhz, medium devsel Memory at fa000000 (32-bit, non-prefetchable) [size=32M] Expansion ROM at f7000000 [disabled] [size=64K]
[ I now see one additional bugs *sigh*: PCI-PCI bridge resources are not properly accounted (4-port Tulip) card. I'll have to fix that. ]
sym53c8xx_2 driver is using IO port space and thus won't get reported in /proc/iomem.
The "stifb" driver claims the upper 16MB of the 32MB BAR and it's all routed to PCI Controller for slot 2 (PCI Bus 4). Since no other devices share that PCI controller, I'm pretty sure the same PCI Controller also gets 0xf100000 range. Add a second gfx card and it's gets uglier. I watched Thomas Boegendorf have fun at LinuxTag by putting 4 similar "Vis-EG" cards in one B2600 and run xinerama. I now wish I had captured the /proc/iomem and lspci -v output for that one!
Anyway, I hope that was an interesting diversion.
cheers,
grant
----- End forwarded message -----
Quoted 3 lines · Grant Grundler
On parisc, this is one of the problems I've dodged.
I'm not looking for a solution right now - but if you know the obvious answer, please tell me. I just want to outline the problem.
Certainly! ;-)
Quoted 5 lines
The PCI controller (
lba_pci.c) on newer PARISC machines can support two MMIO ranges. The "Distributed" range is a "fixed" size and base (1/8th of whatever the parent IO Controller (SBA) has). The second "Directed" Range can use any address otherwise not assigned to PCI devices and directed to any PCI controller under the same SBA.
OK.
Quoted 3 lines
PARISC
lba_pci.ccurrently only advertises Distributed ranges and pretends Directed ranges don't exist. Things still work, but the resource accounting in/proc/iomemis inaccurate.
We can fix that! ;-)
Quoted 5 lines
Since Directed Ranges have precedence in the MMIO address routing, they can overlap with Distributed ranges. Worse, The SBA can assign "duplicate" MMIO address ranges by routing "unused" address space to a Directed range under a different PCI controller. I think both are very ugly scenarios and will need alot of code to detect/handle properly.
I don't think it's such a big problem. Well, I wouldn't, would I? ;-)
Quoted 12 lines
Each Distributed Range is typically only 16MB in size. Consumers of Directed Ranges are things like graphics cards or X.25 cards which want 16MB or greater in a single BAR.
Eg/proc/iomemon a C3000 running 2.4.20 says:
f0000000-f0ffffff : LBA PCI LMMIO
f05d0000-f05d0000 :lcd_data
f05d0008-f05d0008 :lcd_cmd
f1000000-f1ffffff : LBA PCI LMMIO
f4000000-f4ffffff : LBA PCI LMMIO
f4007000-f4007fff : usb-ohci
f4008000-f40083ff : tulip
f4900000-f490007f : tulip
Quoted 8 lines
f4901000-f490107f : tulip
f4902000-f490207f : tulip
f4903000-f490307f : tulip
f6000000-f6ffffff : LBA PCI LMMIO
f6000000-f600ffff : tg3
fa100000-fa4fffff : stifb mmio
fb000000-fbffffff : stifb
...
Thanks for posting the example, it makes life much easier. To take some experience from ccio-based systems, we might favour a /proc/iomem which ends up looking something like this:
f0000000-f0ffffff : PCI bus [10/0/]
f05d0000-f05d0000 : lcd_data
f05d0008-f05d0008 : lcd_cmd
f1000000-f1ffffff : PCI bus [10/1/]
f4000000-f4ffffff : PCI bus [10/2/]
f4007000-f4007fff : usb-ohci
f4008000-f40083ff : tulip
f4900000-f490007f : tulip
f4901000-f490107f : tulip
f4902000-f490207f : tulip
f4903000-f490307f : tulip
f6000000-f6ffffff : PCI bus [10/4/]
f6000000-f600ffff : tg3
fa000000-fbffffff : PCI bus [10/1/]
fa100000-fa4fffff : stifb mmio
fb000000-fbffffff : stifb
So, how to accomplish this? You're right it's going to be a little more tricky than before since we may discover overlapping ranges in any order and the directed range always takes priority over the distributed range.
Calling __request_resource() will return the address of a conflicting resource (if there is one). Only snag is that it's static, but I'm sure no-one will scream too loudly if we unstatic it. So if we're requesting a directed range, we'll tweak the existing range to move it out of our way. If we're requesting a distributed range, we'll shrink our range to avoid colliding. In either case, we will request again.
Quoted 2 lines
[ I now see one additional bugs *sigh*: PCI-PCI bridge resources are not properly accounted (4-port Tulip) card. I'll have to fix that. ]
iomem isn't a device hierarchy... it's not necessary to put _everything_ in. Having said that, I think it'd be kind of nice to have it.
Quoted 8 lines
The "stifb" driver claims the upper 16MB of the 32MB BAR and it's all routed to PCI Controller for slot 2 (PCI Bus 4). Since no other devices share that PCI controller, I'm pretty sure the same PCI Controller also gets
0xf100000range. Add a second gfx card and it's gets uglier. I watched Thomas Boegendorf have fun at LinuxTag by putting 4 similar "Vis-EG" cards in one B2600 and run xinerama. I now wish I had captured the/proc/iomemand lspci -v output for that one!
Hmm.. if I remember rightly, we put a large number of gfx cards in the L2000 in Fort Collins once... wonder if they're still in there.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 5 lines · Matthew Wilcox
Quoted 3 lines
PARISC
lba_pci.ccurrently only advertises Distributed ranges and pretends Directed ranges don't exist. Things still work, but the resource accounting in/proc/iomemis inaccurate.We can fix that! ;-)
yes. ivan also sent me some suggestions (which I suspect are for 2.5). But I still care more about getting 2.4 working right.
Quoted 5 lines
Thanks for posting the example, it makes life much easier. To take some experience from ccio-based systems, we might favour a
/proc/iomemwhich ends up looking something like this:f0000000-f0ffffff : PCI bus [10/0/]
yes - adding the HW path would be a good idea.
It would help identify which LBA/ropes are involved.
But printing the PCI bus number (in addition?) makes more sense since I can then correlate iomem output with lspci output.
Quoted 3 lines
So, how to accomplish this? You're right it's going to be a little more tricky than before since we may discover overlapping ranges in any order and the directed range always takes priority over the distributed range.
exactly. I'm still thinking about it and my current idea is to read all SBA ranges first and then let LBA code resolve conflicts as attemptes to claim ranges. Ivan got me started on that:
| First, you create the set of MMIO resources for SBA which represents | all MMIO ranges. Then you can either shrink or completely remove | those of resources for Distributed ranges which overlap with Directed | range to avoid resource conflicts (this seems to be ok as you says that | the latter has precedence).
The only problem I'm still looking at is Direct Ranges are described by three registers which *only* can be read using 64-bit reads. Tough to do from a (strictly) PA 1.1 built kernel. Still looking at this issue before moving on.
Quoted 6 lines
Calling
__request_resource()will return the address of a conflicting resource (if there is one). Only snag is that it's static, but I'm sure no-one will scream too loudly if we unstatic it. So if we're requesting a directed range, we'll tweak the existing range to move it out of our way. If we're requesting a distributed range, we'll shrink our range to avoid colliding. In either case, we will request again.
yes, this is a good idea. I guess I had the same idea but hadn't thought out how to implement it.
Quoted 5 lines
Quoted 2 lines
[ I now see one additional bugs *sigh*: PCI-PCI bridge resources are not properly accounted (4-port Tulip) card. I'll have to fix that. ]
iomem isn't a device hierarchy... it's not necessary to put _everything_ in. Having said that, I think it'd be kind of nice to have it.
right. /proc/iomem *is* a resource hierarchy and part of the hierarchy is missing in the case of PCI-PCI bridge.
Quoted 2 lines
Hmm.. if I remember rightly, we put a large number of gfx cards in the L2000 in Fort Collins once... wonder if they're still in there.
but L2000 is a PAT PDC box. We'll get different behavior than B2600. Unless PAT PDC supports those as boot devices, none of them will get enabled since lba_pci.c doesn't attempt to program Directed Ranges (aka lba_dev->elmmio).
Anyway, personally, I don't care about gfx cards working under PAT PDC. If someone has to make it work, I'm happy to advise and review patches.
thanks,
grant