PCI bridge range sizing bug
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Adam Jackson
So I'm attempting to do something fairly heinous (X server across five video cards), and I hit a fun bug in bridge range setup. See attached lspci and dmesg, but the short of it is I've got two VGA chips on one card behind a bridge, which is itself behind a second PCI bridge, and the bridge ranges get set up so that I can't map the ROMs, which means I can't post them, and therefore can't use them period.
Quoted 7 lines
The alignment restriction on the ROMs seems a bit extreme:
% sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
% sudo setpci -s 7:2ROM_ADDRESSf0000001(same for 7:1) so that might be part of the problem.
...
Allocating PCI resources starting at 88000000 (gap: 80000000:7ff00000)
...
That's ~2G of space, which should be plenty for your PCI resources I hope? If you have a bunch of cards with large BARS though you might be running out.
...
PCI: Bridge: 0000:00:01.0
IO window: 4000-4fff
MEM window: a3500000-a35fffff (1M)
PREFETCH window: 90000000-97ffffff
PCI: Bridge: 0000:00:03.0
IO window: disabled.
MEM window: a3400000-a34fffff (1M)
PREFETCH window: 98000000-9fffffff
PCI: Bridge: 0000:00:1c.0
IO window: disabled.
MEM window: a3300000-a33fffff (1M)
PREFETCH window: 80000000-8fffffff
PCI: Bridge: 0000:00:1c.4
IO window: 3000-3fff
MEM window: a3200000-a32fffff (1M)
PREFETCH window: a3700000-a37fffff
PCI: Bridge: 0000:00:1c.5
IO window: 2000-2fff
MEM window: a3100000-a31fffff (1M)
PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 0000:07:01.0
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 0000:07:02.0
...
Yep, looks like those two devices had a problem. Supposedly they want to sit at 256M? Given that we're only giving each bridge 1M of memory space that would definitely be a problem.
The total so far is only 5M of PCI space... so we're not making good use of the 2G we were given.
...
PCI: Bridge: 0000:06:00.0
IO window: disabled.
MEM window: a1000000-a2ffffff (32M)
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: 1000-1fff
MEM window: a1000000-a30fffff (~32M)
PREFETCH window: a0000000-a0ffffff
...
And these bridges got more space somehow... Greg who's in charge of our bridge resource allocation code?
Jesse
Quoted 70 lines · Jesse Barnes
Quoted 8 lines · Adam Jackson
So I'm attempting to do something fairly heinous (X server across five video cards), and I hit a fun bug in bridge range setup. See attached lspci and dmesg, but the short of it is I've got two VGA chips on one card behind a bridge, which is itself behind a second PCI bridge, and the bridge ranges get set up so that I can't map the ROMs, which means I can't post them, and therefore can't use them period.
Quoted 7 lines
The alignment restriction on the ROMs seems a bit extreme:
% sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
% sudo setpci -s 7:2ROM_ADDRESSf0000001(same for 7:1) so that might be part of the problem.
...
Allocating PCI resources starting at 88000000 (gap: 80000000:7ff00000)
...That's ~2G of space, which should be plenty for your PCI resources I hope? If you have a bunch of cards with large BARS though you might be running out.
...
PCI: Bridge: 0000:00:01.0
IO window: 4000-4fff
MEM window: a3500000-a35fffff (1M)
PREFETCH window: 90000000-97ffffff
PCI: Bridge: 0000:00:03.0
IO window: disabled.
MEM window: a3400000-a34fffff (1M)
PREFETCH window: 98000000-9fffffff
PCI: Bridge: 0000:00:1c.0
IO window: disabled.
MEM window: a3300000-a33fffff (1M)
PREFETCH window: 80000000-8fffffff
PCI: Bridge: 0000:00:1c.4
IO window: 3000-3fff
MEM window: a3200000-a32fffff (1M)
PREFETCH window: a3700000-a37fffff
PCI: Bridge: 0000:00:1c.5
IO window: 2000-2fff
MEM window: a3100000-a31fffff (1M)
PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 0000:07:01.0
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 0000:07:02.0
...Yep, looks like those two devices had a problem. Supposedly they want to sit at 256M? Given that we're only giving each bridge 1M of memory space that would definitely be a problem.
The total so far is only 5M of PCI space... so we're not making good use of the 2G we were given.
...
PCI: Bridge: 0000:06:00.0
IO window: disabled.
MEM window: a1000000-a2ffffff (32M)
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: 1000-1fff
MEM window: a1000000-a30fffff (~32M)
PREFETCH window: a0000000-a0ffffff
...
Quoted 2 lines
And these bridges got more space somehow... Greg who's in charge of our bridge resource allocation code?
Ivan and Linus seem to be doing the most of the work in this area, I gladly pass these issues on to them :)
thanks,
greg k-h
Quoted 9 lines · Greg KH
Quoted 8 lines · Jesse Barnes
Quoted 7 lines · Adam Jackson
So I'm attempting to do something fairly heinous (X server across five video cards), and I hit a fun bug in bridge range setup. See attached lspci and dmesg, but the short of it is I've got two VGA chips on one card behind a bridge, which is itself behind a second PCI bridge, and the bridge ranges get set up so that I can't map the ROMs, which means I can't post them, and therefore can't use them period.
Ok, let me start out by saying that at this point in the development cycle (ie trying to get 2.6.21 out some day), I can't really find it in myself to care all that deeply about people doing something quote _that_ fairly heinous ;)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The alignment restriction on the ROMs seems a bit extreme:
% sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
% sudo setpci -s 7:2ROM_ADDRESSf0000001
Yeah, they seem to want 256MB.
Quoted 3 lines
Quoted 3 lines
Yep, looks like those two devices had a problem. Supposedly they want to sit at 256M? Given that we're only giving each bridge 1M of memory space that would definitely be a problem.
We should be sizing the bridge regions by how much space the devices behind the bridges actually need, BUT I can guess at two problems:
- On PC's, we generally trust any BIOS setup. If the bridge has been initialized to some value that seems half-way valid we generally leave it there. Moving things around tends to cause a lot more problems than it fixes.
- we normally do *not* try to assign ROM resources at all, because a number of video cards in particular do some really strange stuff with the ROMS, like share the decoders for the ROM's and the other PCI resources!
IOW, I think that what happens is that the BIOS has set it up to have a 32MB window, and since the kernel doesn't think there is anything wrong with that, it leaves it well enough alone.
You could try adding "pci=rom" to the kernel command line, which should make the kernel try to assign space for the roms too.
I think we used to *never* assign PCI bus resources on x86, but that thing got fixed some time ago. Now I think we only re-assign them if they were unassigned *or* if the assignment wasn't working before. But I'm not 100% sure about that second part... It's been working so well that I don't think we've had a lot of problems with resource assignment lately, and I've paged it all out of my brain.
Ivan, can you remind my tired old brain?
Linus
Quoted 8 lines · Linus Torvalds
I think we used to *never* assign PCI bus resources on x86, but that thing got fixed some time ago. Now I think we only re-assign them if they were unassigned *or* if the assignment wasn't working before. But I'm not 100% sure about that second part... It's been working so well that I don't think we've had a lot of problems with resource assignment lately, and I've paged it all out of my brain.
Ivan, can you remind my tired old brain?
No :-) You are absolutely right - we re-assign only unassigned OR conflicting resources. And yes, x86 kernel does accept any enabled bridge ranges without looking at what is on the other side of the bridge.
I think what we need is some very minimalistic validity check for BIOS bridge setup: calculate sums of resource ranges of each type (or just MEM and PREFETCH, should we care about IO these days?) for devices that are behind the bridge, even without taking alignment requirements into account. Then, if some window is too small, we just let the pci_assign_unassigned_resources to take care of that.
Ivan.
Quoted 15 lines · Ivan Kokshaysky
Quoted 8 lines · Linus Torvalds
I think we used to *never* assign PCI bus resources on x86, but that thing got fixed some time ago. Now I think we only re-assign them if they were unassigned *or* if the assignment wasn't working before. But I'm not 100% sure about that second part... It's been working so well that I don't think we've had a lot of problems with resource assignment lately, and I've paged it all out of my brain.
Ivan, can you remind my tired old brain?
No :-) You are absolutely right - we re-assign only unassigned OR conflicting resources. And yes, x86 kernel does accept any enabled bridge ranges without looking at what is on the other side of the bridge.
Quoted 6 lines
I think what we need is some very minimalistic validity check for BIOS bridge setup: calculate sums of resource ranges of each type (or just MEM and PREFETCH, should we care about IO these days?) for devices that are behind the bridge, even without taking alignment requirements into account. Then, if some window is too small, we just let the
pci_assign_unassigned_resourcesto take care of that.
Sounds good, hopefully reassigning the bridge resources won't cause too much trouble. Do you have time to hack this up? If not, I could give it a try, as long as ajax is willing to test...
Thanks,
Jesse
Quoted 4 lines · Jesse Barnes
Sounds good, hopefully reassigning the bridge resources won't cause too much trouble. Do you have time to hack this up? If not, I could give it a try, as long as ajax is willing to test...
Actually, I would suggest we not do it automatically (because the need for it is just so low, and the downsides are potentially huge - there are just too many resources that are "hidden" from us through ACPI tricks and having hardware that doesn't actually expose their PCI resources fully through the normal PCI resource setup).
So I'd suggest that at least initially, we could/should just make it a decision based on a command line flag like the "pci=assign-busses" thing is now (which forces bus *numbers* to be re-assigned regardless of how they were assigned by the firmware, not the actual resources).
Because realistically, I don't think we'll see any common PC platform that actually needs it. Simply because I seriously doubt Windows reassignes bus resources either.
Ivan, want to add some way to force that allocation (something like "pci=assign-bus-resources")
Linus
Quoted 5 lines · Linus Torvalds
Actually, I would suggest we not do it automatically (because the need for it is just so low, and the downsides are potentially huge - there are just too many resources that are "hidden" from us through ACPI tricks and having hardware that doesn't actually expose their PCI resources fully through the normal PCI resource setup).
Definitely. I was intending to enable that *only* with some boot option.
Quoted 2 lines
Ivan, want to add some way to force that allocation (something like "pci=assign-bus-resources")
Yes, hopefully I'll get something in a next couple of days.
Ivan.
Quoted 8 lines · Ivan Kokshaysky
Quoted 6 lines · Linus Torvalds
Actually, I would suggest we not do it automatically (because the need for it is just so low, and the downsides are potentially huge
- there are just too many resources that are "hidden" from us through ACPI tricks and having hardware that doesn't actually expose their PCI resources fully through the normal PCI resource setup).
Quoted 7 lines
Definitely. I was intending to enable that *only* with some boot option.
Quoted 2 lines
Ivan, want to add some way to force that allocation (something like "pci=assign-bus-resources")
Yes, hopefully I'll get something in a next couple of days.
Any update on this, Ivan? Maybe I missed your post, but I haven't seen anything yet...
Thanks,
Jesse
Quoted 2 lines · Jesse Barnes
Any update on this, Ivan? Maybe I missed your post, but I haven't seen anything yet...
No, I didn't post anything, sorry. This turned out to be not as trivial as I thought - I've played with that code on amd64, and results were rather discouraging. For example, I forced reassignment of video card resources and initially this failed because of the way how pci/setup-* allocates ROM
addresses: ROMs are obviously prefetchable, so we're always trying to put them in prefetchable windows of the bridge. Technically, this is correct, but leads to the waste of PCI address space due to size/alignment requirements, especially when ROM is assigned to the same window as a prefetchable framebuffer resource (typically 64-256M).
On the other hand, we could allocate ROMs in non-prefetch ranges just for free, as minimal memory window of the bridges is 1M and both MMIO register blocks and ROMs are usually much smaller.
Probably we need some global variable to control this (and some other things) in pci/setup-*, just like pci_probe on i386...
Another funny thing found on x86_64 - if we ran out of PCI address space below 4G, resources get happily assigned above 4G, even if respective BARs are 32-bit.
Anyway, here's what I have so far, though I doubt that it's a breakthrough for Adam...
Ivan.
arch/i386/pci/common.c +16 -1
@@ -290,6 +290,12 @@ static struct dmi_system_id __devinitdat {} }; +static struct resource pci_mem32 = { + .name ="PCI 32-bit memory space" , + .end =0xffffffff , + .flags =IORESOURCE_MEM , +}; +struct pci_bus *__devinit pcibios_scan_root (int busnum) {struct pci_bus *bus =NULL ;
@@ -305,7 +311,13 @@ struct pci_bus * __devinit pcibios_scan_printk (KERN_DEBUG "PCI: Probing PCI hardware (bus %02x)\n" , busnum); -return pci_scan_bus_parented (NULL , busnum, &pci_root_ops,NULL ); + bus =pci_scan_bus_parented (NULL , busnum, &pci_root_ops,NULL ); +if (bus && bus->resource[1 ] == &iomem_resource) { + pci_mem32.start = pci_mem_start; +if (insert_resource (&iomem_resource, &pci_mem32) ==0 ) + bus->resource[1 ] = &pci_mem32; + } +return bus; }extern u8 pci_cache_line_size;
@@ -418,6 +418,9 @@ char * __devinit pcibios_setup(char *st }else if (!strcmp (str,"routeirq" )) { pci_routeirq =1 ;return NULL ; + }else if (!strcmp (str,"assign-bus-resources" )) { + pci_probe |=PCI_ASSIGN_BUS_RESOURCES ; +return NULL ; }return str; }
arch/i386/pci/pci.h +1 -0
@@ -26,6 +26,7 @@#define PCI_ASSIGN_ROMS 0x1000#define PCI_BIOS_IRQ_SCAN 0x2000#define PCI_ASSIGN_ALL_BUSSES 0x4000 +#define PCI_ASSIGN_BUS_RESOURCES 0x8000extern unsigned int pci_probe;extern unsigned long pirq_table_addr;
arch/i386/pci/i386.c +62 -0
74 lines not on this page · raw message
drivers/pci/setup-bus.c +5 -2
@@ -347,9 +347,12 @@ pbus_size_mem(struct pci_bus *bus, unsigfor (i =0 ; i <PCI_NUM_RESOURCES ; i++) {struct resource *r = &dev->resource[i]; -unsigned long r_size; +unsigned long r_size, r_flags = r->flags; -if (r->parent || (r->flags & mask) != type) +if (i ==PCI_ROM_RESOURCE ) + r_flags &= ~IORESOURCE_PREFETCH ; + +if (r->parent || (r_flags & mask) != type)continue ; r_size = r->end - r->start +1 ;/* For bridges size != alignment */
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Jesse Barnes
Sounds good, hopefully reassigning the bridge resources won't cause too much trouble. Do you have time to hack this up? If not, I could give it a try, as long as ajax is willing to test...
Actually, I would suggest we not do it automatically (because the need for it is just so low, and the downsides are potentially huge - there are just too many resources that are "hidden" from us through ACPI tricks and having hardware that doesn't actually expose their PCI resources fully through the normal PCI resource setup).
Yeah, that's probably prudent. OTOH we should probably let the user know in no uncertain terms that some of the stuff behind one of their bridges will be inaccessible.
Thanks,
Jesse
Quoted 5 lines · Jesse Barnes
Quoted 4 lines · Linus Torvalds
Quoted 3 lines · Jesse Barnes
Sounds good, hopefully reassigning the bridge resources won't cause too much trouble. Do you have time to hack this up? If not, I could give it a try, as long as ajax is willing to test...
Quoted 7 lines
Quoted 5 lines
Actually, I would suggest we not do it automatically (because the need for it is just so low, and the downsides are potentially huge - there are just too many resources that are "hidden" from us through ACPI tricks and having hardware that doesn't actually expose their PCI resources fully through the normal PCI resource setup).
Yeah, that's probably prudent. OTOH we should probably let the user know in no uncertain terms that some of the stuff behind one of their bridges will be inaccessible.
Something like that would have made it a lot more obvious why my Matrox PCIe x1 video card will not work in my Dell 9150, while a PCI video card does work.
The PCI video card directly sits on the bus, and gets its resources assigned by the BIOS.
The PCIe video card turned out to be a PCIe to AGP bridge, and the BIOS did not assign the needed PCI resources, making the system crash when I started X.
X seemed to have some trouble reading the ROM, too...
Signature · 3 lines
Politics is the struggle between those who want to make their country the best in the world, and those who believe it already is. Each group calls the other unpatriotic.