x86: Use larger chunks in mtrr_cleanup
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 x86: Use larger chunks in mtrr_cleanup
Increase the range of chunk sizes tried in mtrr_cleanup() so it is able to map large memory configs into MTRRs.
Currently, mtrr_cleanup() will fail with large memory configurations, because it limits chunk_size to 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.
---
arch/x86/kernel/cpu/mtrr/cleanup.c renamed from arch/x86/kernel/cpu/mtrr/cleanup.c.orig +5 -4
@@ -517,10 +517,11 @@ struct mtrr_cleanup_result {/* * gran_size: 64K, 128K, 256K, 512K, 1M, 2M, ..., 2G -* chunk size: gran_size, ..., 2G -* so we need (1+16)*8 +* chunk size: gran_size, ..., 2G, ..., 1<<address_bits +* (for 32 address bits, we need 136) +* (for 40 address bits, we need 264) */ -#define NUM_RESULT 136 +#define NUM_RESULT 264#define PSHIFT (PAGE_SHIFT - 10)static struct mtrr_cleanup_result __initdata result[NUM_RESULT ];
@@ -751,7 +752,7 @@ int __init mtrr_cleanup(unsigned addressmemset (result,0 ,sizeof (result));for (gran_size = (1ULL <<16 ); gran_size < (1ULL <<32 ); gran_size <<=1 ) { -for (chunk_size = gran_size; chunk_size < (1ULL <<32 ); +for (chunk_size = gran_size; chunk_size < (1ULL <<address_bits); chunk_size <<=1 ) {if (i >=NUM_RESULT )
Quoted 8 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
Luis
Quoted 4 lines · Luis R. Rodriguez
Quoted 3 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.
Quoted 8 lines
Quoted 5 lines
Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do). In any case, Stuart -- could you try booting with 'disable_mtrr_cleanup' as a kernel parameter?
P.
Quoted 13 lines · Prarit Bhargava
Quoted 10 lines · Luis R. Rodriguez
Quoted 8 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.
Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call mtrr_add() directly and if it doesn't then you should be OK.
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?
Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Luis
Quoted 28 lines · Luis R. Rodriguez
Quoted 13 lines · Prarit Bhargava
Quoted 10 lines · Luis R. Rodriguez
Quoted 8 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.
Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it. We can do a stable fix instead to fix older stable kernels.
P.
Quoted 40 lines · Prarit Bhargava
Quoted 28 lines · Luis R. Rodriguez
Quoted 13 lines · Prarit Bhargava
Quoted 10 lines · Luis R. Rodriguez
Quoted 8 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Quoted 3 lines
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it.
Check linux-next, and Documentation/x86/mtrr.txt
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/Documentation/x86/mtrr.txt
The platform use of MTRR is the only thing you should be concerned over but as noted returning MTRR_TYPE_INVALID should suffice if the OS does not make any driver use / modifications.
Another next step I was considering was to then take out PAT initialization code out from MTRR's so that we can enable distros disable MTRR but enable PAT. Right now this is not possible. If there is really a need for MTRR cleanup now's the time to discuss this then, I'd like to know precicely why it'd be needed if the OS does not make any direct use of it.
Quoted 1 line
We can do a stable fix instead to fix older stable kernels.
Patches to stable depend on them being merged first on Linus' tree so we'd need to queue this up for linux-next if you want that, but again, best to first check *why* you need this other than some "some error comes up" type of thing.
Luis
Quoted 46 lines · Luis R. Rodriguez
Quoted 41 lines · Prarit Bhargava
Quoted 29 lines · Luis R. Rodriguez
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Luis R. Rodriguez
Quoted 9 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Quoted 3 lines
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it.
Quoted 8 lines
Check linux-next, and
Documentation/x86/mtrr.txthttps://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/Docume
ntation/x86/mtrr.txtThe platform use of MTRR is the only thing you should be concerned over but as noted returning
MTRR_TYPE_INVALIDshould suffice if the OS does not make any driver use / modifications.
The following sentence in the "mtrr.txt" is not correct. (Sorry I should have caught it earlier.) mtrr_type_lookup() returns MTRR_TYPE_INVALID when MTRRs are disabled, i.e. MTRRs are not set by neither firmware nor OS. Most of the firmwares enable them, though.
"If MTRRs are only set up by the platform firmware code though and the OS does not make any specific MTRR mapping requests mtrr_type_lookup() should always return MTRR_TYPE_INVALID."
Thanks,
-Toshi
Quoted 56 lines · Toshi Kani
Quoted 46 lines · Luis R. Rodriguez
Quoted 41 lines · Prarit Bhargava
Quoted 29 lines · Luis R. Rodriguez
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Luis R. Rodriguez
Quoted 9 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Quoted 3 lines
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it.
Quoted 8 lines
Check linux-next, and
Documentation/x86/mtrr.txthttps://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/Docume
ntation/x86/mtrr.txtThe platform use of MTRR is the only thing you should be concerned over but as noted returning
MTRR_TYPE_INVALIDshould suffice if the OS does not make any driver use / modifications.
Quoted 8 lines
The following sentence in the "mtrr.txt" is not correct. (Sorry I should have caught it earlier.)
mtrr_type_lookup()returnsMTRR_TYPE_INVALIDwhen MTRRs are disabled, i.e. MTRRs are not set by neither firmware nor OS. Most of the firmwares enable them, though."If MTRRs are only set up by the platform firmware code though and the OS does not make any specific MTRR mapping requests
mtrr_type_lookup()should always returnMTRR_TYPE_INVALID."
So this should be clarified to say -- that because platform firmware *may* make use of MTRRs, even if all OS drivers are no longer making use of MTRRs directly, we still need mtrr_type_lookup() to return the right type ?
Luis
Quoted 62 lines · Luis R. Rodriguez
Quoted 61 lines · Toshi Kani
Quoted 46 lines · Luis R. Rodriguez
Quoted 41 lines · Prarit Bhargava
Quoted 29 lines · Luis R. Rodriguez
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Luis R. Rodriguez
Quoted 9 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Quoted 3 lines
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it.
Quoted 8 lines
Check linux-next, and
Documentation/x86/mtrr.txthttps://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/Docume
ntation/x86/mtrr.txtThe platform use of MTRR is the only thing you should be concerned over but as noted returning
MTRR_TYPE_INVALIDshould suffice if the OS does not make any driver use / modifications.The following sentence in the "mtrr.txt" is not correct. (Sorry I should have caught it earlier.)
mtrr_type_lookup()returnsMTRR_TYPE_INVALIDwhen MTRRs are disabled, i.e. MTRRs are not set by neither firmware nor OS. Most of the firmwares enable them, though.
Quoted 7 lines
Quoted 3 lines
"If MTRRs are only set up by the platform firmware code though and the OS does not make any specific MTRR mapping requests
mtrr_type_lookup()should always returnMTRR_TYPE_INVALID."So this should be clarified to say -- that because platform firmware *may* make use of MTRRs, even if all OS drivers are no longer making use of MTRRs directly, we still need
mtrr_type_lookup()to return the right type ?
That is correct, and I think the doc already states that. I'd suggest the following minor changes to the doc:
Thanks,
-Toshi
===
Documentation/x86/mtrr.txt +2 -5
@@ -13,15 +13,12 @@ non-PAT systems while a no-op but equally effective on PAT enabled systems. Even if Linux does not use MTRRs directly, some x86 platform firmware may still set up MTRRs early before booting the OS. They do this as some platform -firmware may still have implemented access to MTRRs which would be controlled -and handled by the platform firmware directly. An example of platform use of +firmware may rely on cache types set by MTRRs. An example of platform use of MTRRs is through the use of SMI handlers, one case could be for fan control, the platform code would need uncachable access to some of its fan control registers. Such platform access does not need any Operating System MTRR code in place other than mtrr_type_lookup() to ensure any OS specific mapping requests -are aligned with platform MTRR setup. If MTRRs are only set up by the platform -firmware code though and the OS does not make any specific MTRR mapping -requests mtrr_type_lookup() should always return MTRR_TYPE_INVALID. +are aligned with platform MTRR setup. For details refer to Documentation/x86/pat.txt.
Quoted 87 lines · Toshi Kani
Quoted 62 lines · Luis R. Rodriguez
Quoted 61 lines · Toshi Kani
Quoted 46 lines · Luis R. Rodriguez
Quoted 41 lines · Prarit Bhargava
Quoted 29 lines · Luis R. Rodriguez
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Luis R. Rodriguez
Quoted 9 lines · Stuart Hayes
Increase the range of chunk sizes tried in
mtrr_cleanup()so it is able to map large memory configs into MTRRs.Currently,
mtrr_cleanup()will fail with large memory configurations, because it limitschunk_sizeto 2GB, which means that each MTRR can only cover 2GB of memory. With a memory size of, say, 256GB, and ten variable MTRRs (such as some recent Intel CPUs have), it is not possible to set up the MTRRs to cover all of memory.Quoted 5 lines
Quoted 2 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do).
There were a few stragglers left on v4.2, I have transformed them in the latest development changes and those tranformations are now part of linux-next. If this is specific to a driver you may want to first ensure you backport the required patch that transforms the driver to use proper PAT interfaces, v4.2 should have most updates but there were still a few left. Just make sure your driver doesn't call
mtrr_add()directly and if it doesn't then you should be OK.Quoted 8 lines
Quoted 2 lines
In any case, Stuart -- could you try booting with
'disable_mtrr_cleanup' as a kernel parameter?Indeed, please I'd like to hear back. Be sure to have the respective driver transformation in place, what driver are you using exactly? In the event that you argue this is still needed I'd like to know exaclty *why*, the comit log does not mention any of that at all.
Quoted 3 lines
Well ... we are trying to also fix this in older kernels too, *cough* RHEL *cough*, so that's where the patch comes from. If upstream is going to deprecate/remove mtrr support so be it.
Quoted 8 lines
Check linux-next, and
Documentation/x86/mtrr.txthttps://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/Docume
ntation/x86/mtrr.txtThe platform use of MTRR is the only thing you should be concerned over but as noted returning
MTRR_TYPE_INVALIDshould suffice if the OS does not make any driver use / modifications.The following sentence in the "mtrr.txt" is not correct. (Sorry I should have caught it earlier.)
mtrr_type_lookup()returnsMTRR_TYPE_INVALIDwhen MTRRs are disabled, i.e. MTRRs are not set by neither firmware nor OS. Most of the firmwares enable them, though.Quoted 7 lines
Quoted 3 lines
"If MTRRs are only set up by the platform firmware code though and the OS does not make any specific MTRR mapping requests
mtrr_type_lookup()should always returnMTRR_TYPE_INVALID."So this should be clarified to say -- that because platform firmware *may* make use of MTRRs, even if all OS drivers are no longer making use of MTRRs directly, we still need
mtrr_type_lookup()to return the right type ?That is correct, and I think the doc already states that. I'd suggest the following minor changes to the doc:
Thanks,
-Toshi===
Documentation/x86/mtrr.txt +0
-1@@ -13,15 +13,12 @@ non-PAT systems while a no-op but equally effective on PAT enabled systems. Even if Linux does not use MTRRs directly, some x86 platform firmware may still set up MTRRs early before booting the OS. They do this as some platform -firmware may still have implemented access to MTRRs which would be controlled
Documentation/x86/mtrr.txt non-PAT systems while a no-op but equally effective on PAT enabled systems. · line 21
Quoted 10 lines
Documentation/x86/mtrr.txt +2
-4-and handled by the platform firmware directly. An example of platform use of +firmware may rely on cache types set by MTRRs. An example of platform use of MTRRs is through the use of SMI handlers, one case could be for fan control, the platform code would need uncachable access to some of its fan control registers. Such platform access does not need any Operating System MTRR code in place other than mtrr_type_lookup() to ensure any OS specific mapping requests -are aligned with platform MTRR setup. If MTRRs are only set up by the platform -firmware code though and the OS does not make any specific MTRR mapping -requests mtrr_type_lookup() should always return MTRR_TYPE_INVALID. +are aligned with platform MTRR setup.
These are still at odds, for instance, I was under the impression we can just have the OS return MTRR_TYPE_INVALID if the OS / drivers never used or set up MTRR, but the platform did, above (not the patch) you seem to be saying that even if the OS didn't modify MTRRs the OS still needs to return the appropriately set up MTRR type by firmware. This is different. Can you clarify?
Luis
On Fri, 2015-09-04 at 00:07 +0200, Luis R. Rodriguez wrote:
:
Quoted 6 lines
These are still at odds, for instance, I was under the impression we can just have the OS return
MTRR_TYPE_INVALIDif the OS / drivers never used or set up MTRR, but the platform did, above (not the patch) you seem to be saying that even if the OS didn't modify MTRRs the OS still needs to return the appropriately set up MTRR type by firmware. This is different. Can you clarify?
mtrr_type_lookup() returns valid MTRR cache type for a given address range when MTRRs are enabled. It does not matter if MTRRs are set by the firmware or the OS. When MTRRs are enabled, the kernel needs to check through mtrr_type_lookup() that large page mapping requests are aligned with MTRRs.
On Xen, or on a platform with firmware that does not enable MTRRs, mtrr_type_lookup() returns MTRR_TYPE_INVALID (as long as the kernel does not enable them).
Thanks,
-Toshi
Quoted 9 lines · Toshi Kani
On Fri, 2015-09-04 at 00:07 +0200, Luis R. Rodriguez wrote:
:Quoted 6 lines
These are still at odds, for instance, I was under the impression we can just have the OS return
MTRR_TYPE_INVALIDif the OS / drivers never used or set up MTRR, but the platform did, above (not the patch) you seem to be saying that even if the OS didn't modify MTRRs the OS still needs to return the appropriately set up MTRR type by firmware. This is different. Can you clarify?
Quoted 4 lines
mtrr_type_lookup()returns valid MTRR cache type for a given address range when MTRRs are enabled. It does not matter if MTRRs are set by the firmware or the OS. When MTRRs are enabled, the kernel needs to check throughmtrr_type_lookup()that large page mapping requests are aligned with MTRRs.
One further change I was considering was seeing if we can separate PAT set up from MTRR's setup, but that was under the assumption we could live with a kernel that would have mtrr_type_lookup() return MTRR_TYPE_INVALID if kernel MTRR code is completely disabled but PAT enabled. We can't enable PAT today without MTRR beceause PAT is initialized from the MTRR init sequence and that depends on MTRR, if we separated these though and and if a distro disabled kernel MTRR an but enabled PAT and if MTRR did set up MTRR what would the possible issues be?
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Quoted 3 lines
or on a platform with firmware that does not enable MTRRs,
mtrr_type_lookup()returnsMTRR_TYPE_INVALID(as long as the kernel does not enable them).
Sure this makes sense.
Thanks in advance,
Luis
Quoted 16 lines · Luis R. Rodriguez
Quoted 9 lines · Toshi Kani
On Fri, 2015-09-04 at 00:07 +0200, Luis R. Rodriguez wrote:
:Quoted 6 lines
These are still at odds, for instance, I was under the impression we can just have the OS return
MTRR_TYPE_INVALIDif the OS / drivers never used or set up MTRR, but the platform did, above (not the patch) you seem to be saying that even if the OS didn't modify MTRRs the OS still needs to return the appropriately set up MTRR type by firmware. This is different. Can you clarify?Quoted 5 lines
mtrr_type_lookup()returns valid MTRR cache type for a given address range when MTRRs are enabled. It does not matter if MTRRs are set by the firmware or the OS. When MTRRs are enabled, the kernel needs to check throughmtrr_type_lookup()that large page mapping requests are aligned with MTRRs.
Quoted 8 lines
One further change I was considering was seeing if we can separate PAT set up from MTRR's setup, but that was under the assumption we could live with a kernel that would have
mtrr_type_lookup()returnMTRR_TYPE_INVALIDif kernel MTRR code is completely disabled but PAT enabled. We can't enable PAT today without MTRR beceause PAT is initialized from the MTRR init sequence and that depends on MTRR, if we separated these though and and if a distro disabled kernel MTRR an but enabled PAT and if MTRR did set up MTRR what would the possible issues be?
PAT's dependency to MTRR could be removed, but I would not recommend disabling the MTRR option since most of the firmwares enable MTRRs. When the kernel has the MTRR option disabled, but the firmware enables MTRRs, the kernel is unable to verify if a large page mapping is aligned with MTRRs. This can lead undefined behavior when such an unaligned map with MTRRs is created and accessed.
Quoted 4 lines
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Right, I meant to say Xen guests. In case of the Xen hypervisor, mtrr_type_lookup() returns a valid type as it runs on a platform.
Thanks,
-Toshi
Quoted 9 lines
Quoted 3 lines
or on a platform with firmware that does not enable MTRRs,
mtrr_type_lookup()returnsMTRR_TYPE_INVALID(as long as the kernel does not enable them).Sure this makes sense.
Thanks in advance,
Luis
Quoted 26 lines · Toshi Kani
Quoted 16 lines · Luis R. Rodriguez
Quoted 9 lines · Toshi Kani
On Fri, 2015-09-04 at 00:07 +0200, Luis R. Rodriguez wrote:
:Quoted 6 lines
These are still at odds, for instance, I was under the impression we can just have the OS return
MTRR_TYPE_INVALIDif the OS / drivers never used or set up MTRR, but the platform did, above (not the patch) you seem to be saying that even if the OS didn't modify MTRRs the OS still needs to return the appropriately set up MTRR type by firmware. This is different. Can you clarify?Quoted 5 lines
mtrr_type_lookup()returns valid MTRR cache type for a given address range when MTRRs are enabled. It does not matter if MTRRs are set by the firmware or the OS. When MTRRs are enabled, the kernel needs to check throughmtrr_type_lookup()that large page mapping requests are aligned with MTRRs.Quoted 8 lines
One further change I was considering was seeing if we can separate PAT set up from MTRR's setup, but that was under the assumption we could live with a kernel that would have
mtrr_type_lookup()returnMTRR_TYPE_INVALIDif kernel MTRR code is completely disabled but PAT enabled. We can't enable PAT today without MTRR beceause PAT is initialized from the MTRR init sequence and that depends on MTRR, if we separated these though and and if a distro disabled kernel MTRR an but enabled PAT and if MTRR did set up MTRR what would the possible issues be?
Quoted 2 lines
PAT's dependency to MTRR could be removed, but I would not recommend disabling the MTRR option since most of the firmwares enable MTRRs.
OK we can put such warning and default to enable MTRR and strongly warn against disabling.
Quoted 5 lines
When the kernel has
the MTRR option disabled, but the firmware enables MTRRs, the kernel is unable to verify if a large page mapping is aligned with MTRRs. This can lead undefined behavior when such an unaligned map with MTRRs is created and accessed.
Crikey!
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Right, I meant to say Xen guests.
Ah but its import complicated than that.
Quoted 2 lines
In case of the Xen hypervisor,
mtrr_type_lookup()returns a valid type as it runs on a platform.
I am not sure if this happens today, I know MTRR is simply disabled by the Xen Hypervisor on the CPU explicitly, it disable it so guests reading the MTRR capabilities sees it as disabled when queried.
Then since the Xen Linux guests cannot speak MTRR through the hypervisor (for instance Xen guests cannot ask Xen hypervisor to mtrr_type_lookup() for it) if PCI passthrough is used it could mean a guest might set up / use incorrect info as well.
If I undestand this correctly then I think we're in a pickle with Xen unless we add hypervisor support and hypercall support for mtrr_type_lookup().
Luis
Quoted 3 lines · Luis R. Rodriguez
Quoted 2 lines · Toshi Kani
Quoted 1 line · Luis R. Rodriguez
On Thu, Sep 03, 2015 at 04:25:31PM -0600, Toshi Kani wrote:
:
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Right, I meant to say Xen guests.
Quoted 8 lines
Ah but its import complicated than that.
Quoted 2 lines
In case of the Xen hypervisor,
mtrr_type_lookup()returns a valid type as it runs on a platform.I am not sure if this happens today, I know MTRR is simply disabled by the Xen Hypervisor on the CPU explicitly, it disable it so guests reading the MTRR capabilities sees it as disabled when queried.
Oh, I would not let the hypervisor to disable MTRRs...
Quoted 7 lines
Then since the Xen Linux guests cannot speak MTRR through the hypervisor (for instance Xen guests cannot ask Xen hypervisor to
mtrr_type_lookup()for it) if PCI passthrough is used it could mean a guest might set up / use incorrect info as well.If I undestand this correctly then I think we're in a pickle with Xen unless we add hypervisor support and hypercall support for
mtrr_type_lookup().
I was under assumption that MTRRs are emulated and disabled on guests. Isn't guest physical address virtualized? I know other proprietary VMMs on IA64, but know nothing about Xen... So, please disregard my comments to Xen. :-)
Thanks,
-Toshi
Quoted 14 lines · Toshi Kani
Quoted 3 lines · Luis R. Rodriguez
Quoted 2 lines · Toshi Kani
Quoted 1 line · Luis R. Rodriguez
On Thu, Sep 03, 2015 at 04:25:31PM -0600, Toshi Kani wrote:
:
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Right, I meant to say Xen guests.
Ah but its import complicated than that.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
In case of the Xen hypervisor,
mtrr_type_lookup()returns a valid type as it runs on a platform.I am not sure if this happens today, I know MTRR is simply disabled by the Xen Hypervisor on the CPU explicitly, it disable it so guests reading the MTRR capabilities sees it as disabled when queried.
Oh, I would not let the hypervisor to disable MTRRs...
Commit 586ab6a055376ec3f3e1e8 ("x86/pvh: disable MTRR feature on cpuid for Dom0") by Roger Pau Monné disables MTRR for PVH dom0, so that cpuid returns that MTRR is disabled to guests. Then later on Linux as of commit 47591df50512 ("xen: Support Xen pv-domains using PAT") added by Juergen as of v3.19 Linux guests can end up booting without MTRR but with PAT now enabled.
Quoted 9 lines
Quoted 7 lines
Then since the Xen Linux guests cannot speak MTRR through the hypervisor (for instance Xen guests cannot ask Xen hypervisor to
mtrr_type_lookup()for it) if PCI passthrough is used it could mean a guest might set up / use incorrect info as well.If I undestand this correctly then I think we're in a pickle with Xen unless we add hypervisor support and hypercall support for
mtrr_type_lookup().I was under assumption that MTRRs are emulated and disabled on guests.
Some "special" flavor Linux guests (with non-upstream code) have guest MTRR hypercall support, for vanilla Xen and Linux they just never get MTRR support. After Juergen's Linux changes though Xen guests can now get shiny PAT support. Since MTRR hypercall support is not upstream and MTRR is ancient I decided instead of adding MTRR hypercall support upstream to go with converting all drivers to PAT interfaces, with the assumption there would be no issues.
Quoted 1 line
Isn't guest physical address virtualized?
It is, there is a xen iotlb and stuff but that should ensure dom0 gets to get proper access to devices, and if you use PCI passthrough you want the best experience as well.
Quoted 2 lines
I know other proprietary VMMs on IA64,
but know nothing about Xen... So, please disregard my comments to Xen. :-)
No worries, no one knows all the answers, we work together to remove cob webs off of these odd corners no one cares about :)
Luis
Quoted 23 lines · Luis R. Rodriguez
Quoted 22 lines · Toshi Kani
Quoted 3 lines · Luis R. Rodriguez
Quoted 2 lines · Toshi Kani
Quoted 1 line · Luis R. Rodriguez
On Thu, Sep 03, 2015 at 04:25:31PM -0600, Toshi Kani wrote:
:
Quoted 16 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
On Xen,
When Xen is used a platform firmware may still set up MTRR, even if the hypervisor doesn't set up MTRR right ? So same issue and question here.
Right, I meant to say Xen guests.
Ah but its import complicated than that.
Quoted 2 lines
In case of the Xen hypervisor,
mtrr_type_lookup()returns a valid type as it runs on a platform.I am not sure if this happens today, I know MTRR is simply disabled by the Xen Hypervisor on the CPU explicitly, it disable it so guests reading the MTRR capabilities sees it as disabled when queried.
Quoted 5 lines
Quoted 1 line
Oh, I would not let the hypervisor to disable MTRRs...
Commit
586ab6a055376ec3f3e1e8("x86/pvh: disable MTRR feature on cpuid for Dom0") by Roger Pau Monné disables MTRR for PVH dom0, so that cpuid returns that MTRR is disabled to guests.
Oh, I see. It just clears the capability bit so that the kernel thinks MTRRs are disabled. That makes sense.
Quoted 3 lines
Then later on Linux as of commit
47591df50512("xen: Support Xen pv-domains using PAT") added by Juergen as of v3.19 Linux guests can end up booting without MTRR but with PAT now enabled.
Nice!
Quoted 25 lines
Quoted 10 lines
Quoted 8 lines
Then since the Xen Linux guests cannot speak MTRR through the hypervisor (for instance Xen guests cannot ask Xen hypervisor to
mtrr_type_lookup()for it) if PCI passthrough is used it could mean a guest might set up / use incorrect info as well.If I undestand this correctly then I think we're in a pickle with Xen unless we add hypervisor support and hypercall support for
mtrr_type_lookup().I was under assumption that MTRRs are emulated and disabled on guests.
Some "special" flavor Linux guests (with non-upstream code) have guest MTRR hypercall support, for vanilla Xen and Linux they just never get MTRR support. After Juergen's Linux changes though Xen guests can now get shiny PAT support. Since MTRR hypercall support is not upstream and MTRR is ancient I decided instead of adding MTRR hypercall support upstream to go with converting all drivers to PAT interfaces, with the assumption there would be no issues.
Quoted 1 line
Isn't guest physical address virtualized?
It is, there is a xen iotlb and stuff but that should ensure dom0 gets to get proper access to devices, and if you use PCI passthrough you want the best experience as well.
Quoted 5 lines
Quoted 2 lines
I know other proprietary VMMs on IA64, but know nothing about Xen... So, please disregard my comments to Xen. :-)
No worries, no one knows all the answers, we work together to remove cob webs off of these odd corners no one cares about :)
Thanks for all the info! That helps.
-Toshi
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 04.09.15 at 01:54, <[email protected]> wrote:
If I undestand this correctly then I think we're in a pickle with Xen unless we add hypervisor support and hypercall support for
mtrr_type_lookup().
Are you perhaps unaware of XENPF_read_memtype (which our kernel uses)?
Jan
Quoted 11 lines
Quoted 4 lines
Linux drivers no longer use MTRR so why is the cleanup needed, ie, what would happen if the cleanup is just skipped in your case ?
The infiniband & video drivers still use MTRR (or at least it was my understanding that they do). In any case, Stuart -- could you try booting with '
disable_mtrr_cleanup' as a kernel parameter?P.
Sorry for the delayed response.
Booting with 'disable_mtrr_cleanup' works, but the system I am working with isn't actually failing--it just gets ugly error messages. And the BIOS on the system I am working with had set up the MTRRs correctly.
Stuart
Quoted 4 lines · Stuart Hayes
Booting with '
disable_mtrr_cleanup' works, but the system I am working with isn't actually failing--it just gets ugly error messages. And the BIOS on the system I am working with had set up the MTRRs correctly.
Please post boot log and /proc/mtrr for:
1. without your patch
2. without your patch and with disable_mtrr_cleanup in boot command line.
3. with your patch.
Thanks
Yinghai
Quoted 10 lines · Yinghai Lu
Quoted 4 lines · Stuart Hayes
Booting with '
disable_mtrr_cleanup' works, but the system I am working with isn't actually failing--it just gets ugly error messages. And the BIOS on the system I am working with had set up the MTRRs correctly.Please post boot log and
/proc/mtrrfor:
1. without your patch
2. without your patch and withdisable_mtrr_cleanupin boot command line.
3. with your patch.
Stuart,
to provide some context -- I reached out to Yinghai as he wrote the original mtrr cleanup code. The commit logs seem to read that a crash was possible on systems with > 4 GiB RAM with some types of BIOSes... The cleanup code seems to trigger when variable MTRRs do not exist that are UC, or when all varible MTRRs that exist are just UC + WB (Yinghai correct me if I'm wrong). The commit log in question (95ffa2438d0e9 "x86: mtrr cleanup for converting continuous to discrete layout, v8") was not very clear about the cause of the crash -- but suppose the issue here was the BIOS on some systems might want to create some UC variable MTRRs early on and there was no UC MTRRs available, and I can only guess the cleanup exists as hack for those BIOSes. Even if that was the case -- its still not clear *why* the crash would happen but I suppose a driver mishap can happen without UC guarantees for some devices the BIOS may want to enable UC MTRR on.
To be able to determine what we do upstream we need to understand the above first. We also need to understand if the cleanup might also be implicated by userspace drivers using /proc/mtrr, or if a proprietary driver exists that does use mtrr_add() directly even though PAT has been available for ages and all drivers are now properly converted.
With clear answers to the above we'll be able to determine what the right course of action should be for this patch. For instance I'm inclined to strive to disable the complex cleanup code if we don't need it anymore, but if we do need it your patch makes sense. If the patch makes sense then though are we going to have to keep updating the segment size *every time* as systems grow? That seems rather silly. And if PAT is prevalent why are vendors adding MTRRs still? The cleanup seems complex and a major hack for a fix for some BIOSes, I'd much rather identify the exact issue and only have a fix to address that case.
Luis
Quoted 35 lines · Luis R. Rodriguez
Quoted 10 lines · Yinghai Lu
Quoted 4 lines · Stuart Hayes
Booting with '
disable_mtrr_cleanup' works, but the system I am working with isn't actually failing--it just gets ugly error messages. And the BIOS on the system I am working with had set up the MTRRs correctly.Please post boot log and
/proc/mtrrfor:
1. without your patch
2. without your patch and withdisable_mtrr_cleanupin boot command line.
3. with your patch.Stuart,
to provide some context -- I reached out to Yinghai as he wrote the original mtrr cleanup code. The commit logs seem to read that a crash was possible on systems with > 4 GiB RAM with some types of BIOSes... The cleanup code seems to trigger when variable MTRRs do not exist that are UC, or when all varible MTRRs that exist are just UC + WB (Yinghai correct me if I'm wrong). The commit log in question (
95ffa2438d0e9"x86: mtrr cleanup for converting continuous to discrete layout, v8") was not very clear about the cause of the crash -- but suppose the issue here was the BIOS on some systems might want to create some UC variable MTRRs early on and there was no UC MTRRs available, and I can only guess the cleanup exists as hack for those BIOSes. Even if that was the case -- its still not clear *why* the crash would happen but I suppose a driver mishap can happen without UC guarantees for some devices the BIOS may want to enable UC MTRR on.To be able to determine what we do upstream we need to understand the above first. We also need to understand if the cleanup might also be implicated by userspace drivers using
/proc/mtrr, or if a proprietary driver exists that does usemtrr_add()directly even though PAT has been available for ages and all drivers are now properly converted.
Quoted 10 lines
With clear answers to the above we'll be able to determine what the right course of action should be for this patch. For instance I'm inclined to strive to disable the complex cleanup code if we don't need it anymore, but if we do need it your patch makes sense. If the patch makes sense then though are we going to have to keep updating the segment size *every time* as systems grow? That seems rather silly. And if PAT is prevalent why are vendors adding MTRRs still? The cleanup seems complex and a major hack for a fix for some BIOSes, I'd much rather identify the exact issue and only have a fix to address that case.
I never heard back... so let's take this up on the other thread I just raised.
Luis
Quoted 47 lines · Luis R. Rodriguez
Quoted 35 lines · Luis R. Rodriguez
Quoted 10 lines · Yinghai Lu
Quoted 4 lines · Stuart Hayes
Booting with '
disable_mtrr_cleanup' works, but the system I am working with isn't actually failing--it just gets ugly error messages. And the BIOS on the system I am working with had set up the MTRRs correctly.Please post boot log and
/proc/mtrrfor:
1. without your patch
2. without your patch and withdisable_mtrr_cleanupin boot command line.
3. with your patch.Stuart,
to provide some context -- I reached out to Yinghai as he wrote the original mtrr cleanup code. The commit logs seem to read that a crash was possible on systems with > 4 GiB RAM with some types of BIOSes... The cleanup code seems to trigger when variable MTRRs do not exist that are UC, or when all varible MTRRs that exist are just UC + WB (Yinghai correct me if I'm wrong). The commit log in question (
95ffa2438d0e9"x86: mtrr cleanup for converting continuous to discrete layout, v8") was not very clear about the cause of the crash -- but suppose the issue here was the BIOS on some systems might want to create some UC variable MTRRs early on and there was no UC MTRRs available, and I can only guess the cleanup exists as hack for those BIOSes. Even if that was the case -- its still not clear *why* the crash would happen but I suppose a driver mishap can happen without UC guarantees for some devices the BIOS may want to enable UC MTRR on.To be able to determine what we do upstream we need to understand the above first. We also need to understand if the cleanup might also be implicated by userspace drivers using
/proc/mtrr, or if a proprietary driver exists that does usemtrr_add()directly even though PAT has been available for ages and all drivers are now properly converted.Quoted 10 lines
With clear answers to the above we'll be able to determine what the right course of action should be for this patch. For instance I'm inclined to strive to disable the complex cleanup code if we don't need it anymore, but if we do need it your patch makes sense. If the patch makes sense then though are we going to have to keep updating the segment size *every time* as systems grow? That seems rather silly. And if PAT is prevalent why are vendors adding MTRRs still? The cleanup seems complex and a major hack for a fix for some BIOSes, I'd much rather identify the exact issue and only have a fix to address that case.
Quoted 2 lines
I never heard back... so let's take this up on the other thread I just raised.
Although we never got an answer, it seems now its clear, at least Toshi has provided a confirmation that the cleanup is no longer needed given that "cleanup was needed to allocate more free slots to MTRRs. We do not need to care about the number of free slots as long as the kernel does not insert any new entry to MTRRs" [0]. So instead of enhancing the cleanup code for larger systems we should now just remove this cleanup code completely now.
[0] http://lkml.kernel.org/r/[email protected]
Luis