PCI extended conf space when MMCONFIG disabled because of e820
Hi Arjan,
We have some machines here where MMCONFIG is disabled in 2.6.17 because their MCFG area is not e820-reserved. It makes the extended PCI config space unavailable to pci_read/write_config_foo(). I don't know if lots of people out there use the extended config space, but at least we do in our myri10ge driver.
What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.
Actually, this problem seems to target nVidia chipsets, and we actually know how to check this chipset's registers to be sure whether MMCONFIG works. So it might be good to improve the current MMCONFIG disabling code by adding some chipset-specific hacks (having nVidia and Intel chipsets there may cover most of the cases). I don't think there's any way to do that with PCI quirks. We might end up having these hacks in the MMCONFIG initialization code.
Thanks,
Brice
Brice Goglin wrote:er.
Quoted 8 lines
What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.
an OS isn't allowed to mix old and new access methods realistically so I don't think this is a viable good solution...
Quoted 3 lines · Arjan van de Ven
Brice Goglin wrote:er.
Quoted 2 lines
What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
One problem in your proposal Bryan is that just trying to access bogus hardware mappings might cause nasty effects like machine checks or trigger chipset errata.
Quoted 3 lines
Quoted 3 lines
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
That's already the case.
Quoted 2 lines
Quoted 2 lines
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.
(3) sounds dangerous to me.
Quoted 2 lines
an OS isn't allowed to mix old and new access methods realistically so I don't think this is a viable good solution...
Why is it not allowed? We already do it in some cases.
Anyways I would say that if the BIOS can't get MCFG right then it's likely not been validated on that board and shouldn't be used.
-Andi
Quoted 10 lines · Arjan van de Ven
Brice Goglin wrote:er.
Quoted 8 lines
What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.
Quoted 3 lines
an OS isn't allowed to mix old and new access methods realistically so I don't think
this is a viable good solution...
Well, we are talking about using a different method to access the extended config space only. This space is independent from the legacy config space.
I don't see how mixing the old and new methods like this could lead to any problem, we are not going to mix them to access the same registers. But I agree with Andi about the possible dangerousness.
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :) If you still do not like this first proposal, what do you think of my other one ? (having chipset-specific checks in pci_mmcfg_init to find out for sure whether mmconfig will work)
Thanks,
Brice
Quoted 4 lines
Well, we are talking about using a different method to access the extended config space only. This space is independent from the legacy config space.
not really. In many ways it's the same space.
Quoted 2 lines
I don't see how mixing the old and new methods like this could lead to any problem, we are not going to mix them to access the same registers.
the spec simply doesn't allow it. Sure it may work on your machine today, but that doesn't make it a good idea ;)
Quoted 3 lines
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)
it's unlikely to be many machines though.
If you still do not like this first proposal, what do you
Quoted 2 lines
think of my other one ? (having chipset-specific checks in
pci_mmcfg_initto find out for sure whether mmconfig will work)
I'm all in favor of a more detailed test; just we HAVE to have a test for this since it's simply broken too often. What the test needs to do is check if the MCFG entry actually points to a working mmconfig area, so 1) that it actually points to an mmconfig area and not to garbage, and 2) that accesses to it actually work ;)
the current approach doesn't test 2) realistically, only 1), but if you weaken the 1) test as you propose you really ought to substitute it with a 2) test...
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)
it's unlikely to be many machines though.
I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing. Digging a bit deeper, I found this in the PCI firmware spec (v 3.0):
If the operating system does not natively comprehend reserving the MMCFG region, the MMCFG region must be reserved by firmware. The address range reported in the MCFG table or by _CBA method (see Section 4.1.3) must be reserved by declaring a motherboard resource. For most systems, the motherboard resource would appear at the root of the ACPI namespace (under \_SB) in a node with a _HID of EISAID (PNP0C02), and the resources in this case should not be claimed in the root PCI bus.s _CRS. The resources can optionally be returned in Int15 E820 or EFIGetMemoryMap as reserved memory but must always be reported through ACPI as a motherboard resource
Sure enough, the ACPI namespace for the "broken" machines lists the MMCFG resources as indicated above, and PCI Express works fine otherwise. I haven't looked yet whether it's possible to add this check in the code, have you looked into that option? I understand the PCI firmware spec is not necessarily the final authority on this, but a _lot_ of BIOS developers read that to figure out what to do...
Rajesh
Quoted 10 lines · Rajesh Shah
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)
it's unlikely to be many machines though.
I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.
What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.
Error handling seems to be still a quite obscure feature, not used by many people - so booting is more important than supporting it. Still would be good to support it of course.
You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?
Is it partially reserved (not for all busses) or not at all?
Quoted 7 lines
Sure enough, the ACPI namespace for the "broken" machines lists the MMCFG resources as indicated above, and PCI Express works fine otherwise. I haven't looked yet whether it's possible to add this check in the code, have you looked into that option? I understand the PCI firmware spec is not necessarily the final authority on this, but a _lot_ of BIOS developers read that to figure out what to do...
If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.
The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.
-Andi
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Rajesh Shah
Quoted 5 lines · Arjan van de Ven
Quoted 4 lines
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
it's unlikely to be many machines though.
I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.
What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.
Yes, I meant it killed extended config space. Note that we are about to send out code that enables PCI Express error handling, so this will become more visible now.
Quoted 3 lines
You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?
Yes, I saw this in 3 out of 4 systems I checked. I'll go check some more systems too.
Quoted 2 lines
Is it partially reserved (not for all busses) or not at all?
The MMCFG resources are not listed at all in the BIOS provided memory map.
Quoted 7 lines
If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.
The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.
Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.
Rajesh
Quoted 33 lines · Rajesh Shah
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Rajesh Shah
Quoted 3 lines · Arjan van de Ven
Quoted 3 lines
We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
it's unlikely to be many machines though.
I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.
What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.
Yes, I meant it killed extended config space. Note that we are about to send out code that enables PCI Express error handling, so this will become more visible now.
Quoted 3 lines
You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?
Yes, I saw this in 3 out of 4 systems I checked. I'll go check some more systems too.
Quoted 2 lines
Is it partially reserved (not for all busses) or not at all?
The MMCFG resources are not listed at all in the BIOS provided memory map.
Quoted 5 lines
If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.
The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.
Quoted 7 lines
Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.
well... it's sort of common sense though.. if you want non-ACPI OSes to work properly (like the older 2.4 based distros...)
Quoted 9 lines · Arjan van de Ven
Quoted 7 lines · Rajesh Shah
Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.
Quoted 3 lines
well... it's sort of common sense though.. if you want non-ACPI OSes to work properly (like the older 2.4 based distros...)
In this case we already have an ACPI dependence, since the MCFG table is listed in ACPI. In any case, I have a patch that got me my extended config space back on the one machine I've tested so far. I'll test it out on the remaining 2 where I saw this problem and send it out.
Rajesh