Skip to contentSkip to conversation

PCI extended conf space when MMCONFIG disabled because of e820

RFC Brice Goglin 4 participants 10 messages Jun 14, 2006 – Jun 23, 2006

lore
Share & tools
lore
#0 lore raw

Hi Arjan,

We have some machines here where MMCONFIG is disabled in 2.6.17 because their MCFG area is not e820-reserved. It makes the extended PCI config space unavailable to pci_read/write_config_foo(). I don't know if lots of people out there use the extended config space, but at least we do in our myri10ge driver.

What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.

Actually, this problem seems to target nVidia chipsets, and we actually know how to check this chipset's registers to be sure whether MMCONFIG works. So it might be good to improve the current MMCONFIG disabling code by adding some chipset-specific hacks (having nVidia and Intel chipsets there may cover most of the cases). I don't think there's any way to do that with PCI quirks. We might end up having these hacks in the MMCONFIG initialization code.

Thanks,
Brice

#1 lore raw

Brice Goglin wrote:er.

Quoted 8 lines

What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.

an OS isn't allowed to mix old and new access methods realistically so I don't think this is a viable good solution...

#2 lore raw
Quoted 3 lines · Arjan van de Ven

Brice Goglin wrote:er.

Quoted 2 lines

What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)

One problem in your proposal Bryan is that just trying to access bogus hardware mappings might cause nasty effects like machine checks or trigger chipset errata.

Quoted 3 lines
Quoted 3 lines

2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)

That's already the case.

Quoted 2 lines
Quoted 2 lines

3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.

(3) sounds dangerous to me.

Quoted 2 lines

an OS isn't allowed to mix old and new access methods realistically so I don't think this is a viable good solution...

Why is it not allowed? We already do it in some cases.

Anyways I would say that if the BIOS can't get MCFG right then it's likely not been validated on that board and shouldn't be used.

-Andi

#3 lore raw
In reply to Arjan van de Ven
Quoted 10 lines · Arjan van de Ven

Brice Goglin wrote:er.

Quoted 8 lines

What would you think of a patch implementing the following strategy:
1) if MMCONFIG works, always use it (no change)
2) if MMCONFIG is disabled and we are accessing the regular config space, use direct conf (no change, should ensure that any machine will still boot fine)
3) if MMCONFIG is disabled but we are accessing the _extended_ config space, try mmconfig anyway since there's no other way to do it.

Quoted 3 lines

an OS isn't allowed to mix old and new access methods realistically so I don't think
this is a viable good solution...

Well, we are talking about using a different method to access the extended config space only. This space is independent from the legacy config space.
I don't see how mixing the old and new methods like this could lead to any problem, we are not going to mix them to access the same registers. But I agree with Andi about the possible dangerousness.

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :) If you still do not like this first proposal, what do you think of my other one ? (having chipset-specific checks in pci_mmcfg_init to find out for sure whether mmconfig will work)

Thanks,
Brice

#4 lore raw
Quoted 4 lines

Well, we are talking about using a different method to access the extended config space only. This space is independent from the legacy config space.

not really. In many ways it's the same space.

Quoted 2 lines

I don't see how mixing the old and new methods like this could lead to any problem, we are not going to mix them to access the same registers.

the spec simply doesn't allow it. Sure it may work on your machine today, but that doesn't make it a good idea ;)

Quoted 3 lines

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)

it's unlikely to be many machines though.

If you still do not like this first proposal, what do you

Quoted 2 lines

think of my other one ? (having chipset-specific checks in pci_mmcfg_init to find out for sure whether mmconfig will work)

I'm all in favor of a more detailed test; just we HAVE to have a test for this since it's simply broken too often. What the test needs to do is check if the MCFG entry actually points to a working mmconfig area, so 1) that it actually points to an mmconfig area and not to garbage, and 2) that accesses to it actually work ;)

the current approach doesn't test 2) realistically, only 1), but if you weaken the 1) test as you propose you really ought to substitute it with a 2) test...

#5 lore raw
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)

it's unlikely to be many machines though.

I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing. Digging a bit deeper, I found this in the PCI firmware spec (v 3.0):

If the operating system does not natively comprehend reserving the MMCFG region, the MMCFG region must be reserved by firmware. The address range reported in the MCFG table or by _CBA method (see Section 4.1.3) must be reserved by declaring a motherboard resource. For most systems, the motherboard resource would appear at the root of the ACPI namespace (under \_SB) in a node with a _HID of EISAID (PNP0C02), and the resources in this case should not be claimed in the root PCI bus.s _CRS. The resources can optionally be returned in Int15 E820 or EFIGetMemoryMap as reserved memory but must always be reported through ACPI as a motherboard resource

Sure enough, the ACPI namespace for the "broken" machines lists the MMCFG resources as indicated above, and PCI Express works fine otherwise. I haven't looked yet whether it's possible to add this check in the code, have you looked into that option? I understand the PCI firmware spec is not necessarily the final authority on this, but a _lot_ of BIOS developers read that to figure out what to do...

Rajesh

#6 lore raw
Quoted 10 lines · Rajesh Shah
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)

it's unlikely to be many machines though.

I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.

What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.

Error handling seems to be still a quite obscure feature, not used by many people - so booting is more important than supporting it. Still would be good to support it of course.

You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?

Is it partially reserved (not for all busses) or not at all?

Quoted 7 lines

Sure enough, the ACPI namespace for the "broken" machines lists the MMCFG resources as indicated above, and PCI Express works fine otherwise. I haven't looked yet whether it's possible to add this check in the code, have you looked into that option? I understand the PCI firmware spec is not necessarily the final authority on this, but a _lot_ of BIOS developers read that to figure out what to do...

If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.

The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.

-Andi

#7 lore raw
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Rajesh Shah
Quoted 5 lines · Arjan van de Ven
Quoted 4 lines

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

it's unlikely to be many machines though.

I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.

What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.

Yes, I meant it killed extended config space. Note that we are about to send out code that enables PCI Express error handling, so this will become more visible now.

Quoted 3 lines

You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?

Yes, I saw this in 3 out of 4 systems I checked. I'll go check some more systems too.

Quoted 2 lines

Is it partially reserved (not for all busses) or not at all?

The MMCFG resources are not listed at all in the BIOS provided memory map.

Quoted 7 lines

If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.

The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.

Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.

Rajesh

#8 lore raw
Quoted 33 lines · Rajesh Shah
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Rajesh Shah
Quoted 3 lines · Arjan van de Ven
Quoted 3 lines

We need to improve this "mmconfig disabled" anyhow. Having the extended config space unavailable on lots of machines is also far from a viable solution :)

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

it's unlikely to be many machines though.

I just noticed today - this check killed PCI Express on 3 of the 4 machines I normally use for testing.

What do you mean with killed PCI Express? PCI Express should work even without extended config space, except error handling.

Yes, I meant it killed extended config space. Note that we are about to send out code that enables PCI Express error handling, so this will become more visible now.

Quoted 3 lines

You're saying that you have lots of machines where the mmconfig aperture is not fully reserved in e820?

Yes, I saw this in 3 out of 4 systems I checked. I'll go check some more systems too.

Quoted 2 lines

Is it partially reserved (not for all busses) or not at all?

The MMCFG resources are not listed at all in the BIOS provided memory map.

Quoted 5 lines

If someone does a patch to double check it against the ACPI name space I'm not opposed to let it overrule the e820 heuristic.

The point of this code is to pragmatically detect BIOS with obviously broken setups. It's not about standards lawyering.

Quoted 7 lines

Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.

well... it's sort of common sense though.. if you want non-ACPI OSes to work properly (like the older 2.4 based distros...)

#9 lore raw
Quoted 9 lines · Arjan van de Ven
Quoted 7 lines · Rajesh Shah

Oh I agree with you that booting is more important. My point with the spec statement was that most BIOS developers may not even know they are doing something "wrong" by not listing these resources in the int15 E820 table, since the document they normally refer to doesn't say so. I suspect there are many more systems out there which do the same thing and will fail the check, but we never notice since most users don't try to ever access the extended space today.

Quoted 3 lines

well... it's sort of common sense though.. if you want non-ACPI OSes to work properly (like the older 2.4 based distros...)

In this case we already have an ACPI dependence, since the MCFG table is listed in ACPI. In any case, I have a patch that got me my extended config space back on the one machine I've tested so far. I'll test it out on the remaining 2 where I saw this problem and send it out.

Rajesh