MTRR initialization
Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
The X server and various other programs try to add a mapping for my video card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this always fails with a type mismatch error (old type is write-back). Apparently it's conflicting with mapping register 0. I can't just disable the existing settings and re-add them; the system hangs soon after disabling reg01.
I guess the kernel must be getting the initial setup from the BIOS. I've hacked around this in mtrr/generic.c by explicitly changing the MTRR state in get_mtrr_state to split the first mapping into two; one at base 0 size 2048M and one at base 2048M size 1024M. So now I have this, which is pretty much what I wanted:
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
So the question is - was there an easier/correct way to do this?
It might have been nice if the MTRR ioctls allowed the register number to be specified on the Set commands, though I'm not sure that would have helped in this case.
Signature · 4 lines
-- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Quoted 8 lines · Howard Chu
Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
BIOS should have another setup option about MTRR (continous?) then you could get
Quoted 2 lines
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1 ===> if you have rev E...instead of rev F.
YH
Quoted 10 lines · Yinghai Lu
Quoted 8 lines · Howard Chu
Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
Quoted 7 lines
BIOS should have another setup option about MTRR (continous?) then you could get
Quoted 2 lines
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1 ===> if you have rev E...instead of rev F.
Thanks for the reply. Unfortunately this BIOS doesn't seem to have any other options related to the MTRRs. I guess I'll just live with this hack.
Signature · 4 lines
-- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
generic.c renamed from generic.c.O +17 -0
@@ -78,6 +78,8 @@ base, base + step - 1, mtrr_attrib_to_str(*types)); } +static int set_mtrr_var_ranges (unsigned int index,struct mtrr_var_range *vr); +/* Grab all of the MTRR state for this CPU into *state */ void get_mtrr_state (void ) {
@@ -105,6 +107,21 @@ mtrr_state.def_type = (lo &0xff ); mtrr_state.enabled = (lo &0xc00 ) >>10 ; +/* Check for initial 4GB range, it should only be 3GB */ +if (!mtrr_state.var_ranges[0 ].base_hi && !(mtrr_state.var_ranges[0 ].base_lo &0xffffff00 ) && + !(mtrr_state.var_ranges[0 ].mask_lo &0x80000000 )) { +/* split initial 4GB range into 2GB and 1GB */ +for (i = num_var_ranges-1 ; i>0 ; i--) + mtrr_state.var_ranges[i] = mtrr_state.var_ranges[i-1 ]; + mtrr_state.var_ranges[0 ].mask_lo |=0x80000000 ; + mtrr_state.var_ranges[1 ].base_hi =0 ; + mtrr_state.var_ranges[1 ].base_lo =0x80000000 |MTRR_TYPE_WRBACK ; + mtrr_state.var_ranges[1 ].mask_hi = mtrr_state.var_ranges[2 ].mask_hi; + mtrr_state.var_ranges[1 ].mask_lo = mtrr_state.var_ranges[2 ].mask_lo; +for (i =0 ; i < num_var_ranges; i++) +set_mtrr_var_ranges (i, &mtrr_state.var_ranges[i]); + } +if (mtrr_show) {int high_width;
Quoted 10 lines · Howard Chu
Quoted 9 lines · Yinghai Lu
Quoted 8 lines · Howard Chu
Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
and you will have problem...
So good way is get the ram top, and hole size...and reset the whole set ot var mtrr.
you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part about CONFIG_VAR_MTRR_HOLE...
but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.
YH
Quoted 23 lines · Yinghai Lu
Quoted 3 lines · Howard Chu
Quoted 2 lines · Yinghai Lu
Quoted 1 line · Howard Chu
Hi, was wondering if anyone else has been tripped up by this... I've got
4GB of
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1and you will have problem...
So good way is get the ram top, and hole size...and reset the whole set ot var mtrr.
Quoted 6 lines
you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part aboutCONFIG_VAR_MTRR_HOLE...but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.
There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.
So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.
There is some difficulty there but software can always do those things safely.
Eric
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines
but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.
There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.
Well the BIOS is definitely doing it wrong here. As I mentioned before, it was setting up
0-0x100000000 WB 0xc0000000 - 0x100000000 UC 0xc0000000 - 0xd0000000 WC
But the Intel Architecture Software Developer's Manual states that whenever any variable MTRR range overlaps with an UC MTRR range, the range remains UC. (Section 9.12.2.3). So in fact what I needed to set was
0-2GB WB 2-3GB WB 3-3.25GB WC
and delete the 3-4GB UC range to get the behavior that the BIOS seems to have been intending to set up. (Relying on the default of UC for the unspecified ranges.)
Quoted 7 lines
So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.There is some difficulty there but software can always do those things safely.
Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always UC regardless of the bits in the page table. So with the MTRR setup that the BIOS left me with, this is still a no-go. There's no way to get the desired effect without completely reinitializing the MTRRs.
Of course, this isn't the only problem with these Asus BIOSs...
Signature · 4 lines
-- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Quoted 10 lines · Howard Chu
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines
but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.
There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.
Quoted 5 lines
Well the BIOS is definitely doing it wrong here. As I mentioned before, it was setting up
0-0x100000000 WB 0xc0000000 - 0x100000000 UC 0xc0000000 - 0xd0000000 WC
Ok. I do agree that is wrong, and really weird to.
Quoted 9 lines
But the Intel Architecture Software Developer's Manual states that whenever any variable MTRR range overlaps with an UC MTRR range, the range remains UC. (Section 9.12.2.3). So in fact what I needed to set was
0-2GB WB 2-3GB WB 3-3.25GB WCand delete the 3-4GB UC range to get the behavior that the BIOS seems to have been intending to set up. (Relying on the default of UC for the unspecified ranges.)
Which is certainly a good way to go if your hole sizes allow and you have enough MTRRs.
Quoted 5 lines
Quoted 5 lines
So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.
Quoted 7 lines
Quoted 2 lines
There is some difficulty there but software can always do those things safely.
Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always UC regardless of the bits in the page table. So with the MTRR setup that the BIOS left me with, this is still a no-go. There's no way to get the desired effect without completely reinitializing the MTRRs.
Please look a little more closely. WC is a flavor of UC. There is a specific exception that allows page tables to promote UC to WC.
I don't seem to have the same version of this document so I can't refer you to specific sections. But I have looked it up several times and every time I have looked I have found the exception that allows PAT to promote UC to WC.
Quoted 1 line
Of course, this isn't the only problem with these Asus BIOSs...
Sure.
My point was that when we try to solve the general problem rather then a specific case there are some very practical limits on what can be done.
Now it should be mentioned that you can go in with /proc/mtrr and fix things manually on a specific machine.
Eric
Quoted 5 lines · Howard Chu
So the question is - was there an easier/correct way to do this?
It might have been nice if the MTRR ioctls allowed the register number to be specified on the Set commands, though I'm not sure that would have helped in this case.
To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...
Jesse
Quoted 5 lines · Jesse Barnes
To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...
PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 2 lines · Jesse Barnes
To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...
PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.
Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?
On my home machine for instance, the BIOS uses all 8 MTRRs leaving none for X. I hacked it by merging a couple of MTRRs but this isn't a generic solution.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 2 lines
Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?
That risks breaking SMM or BIOS code.
-Andi
Quoted 6 lines · Andi Kleen
Quoted 2 lines
Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?
That risks breaking SMM or BIOS code.
Yes. Too bad.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Jesse Barnes
To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...
PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.
Look at the original bug report. There's a bug WB region so any subsequent WC mappings via the MTRRs won't work, so you need PAT to do WC (unless you want to rewrite all your MTRR settings as Howard has to do now).
Seems like more and more machines are being setup this way, and until we have PAT users will have to jump through hoops to get good performance.
Jesse
Quoted 11 lines · Jesse Barnes
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Jesse Barnes
To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...
PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.
Look at the original bug report. There's a bug WB region so any
^^^
err bigQuoted 27 lines · Howard Chu
Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1The X server and various other programs try to add a mapping for my video card's buffer, at
0xd0000000, size=256MB, type=write-combining, and this always
fails with a type mismatch error (old type is write-back). Apparently it's conflicting with mapping register 0. I can't just disable the existing settings
and re-add them; the system hangs soon after disabling reg01.I guess the kernel must be getting the initial setup from the BIOS. I've hacked
around this inmtrr/generic.cby explicitly changing the MTRR state inget_mtrr_stateto split the first mapping into two; one at base 0 size 2048M
and one at base 2048M size 1024M. So now I have this, which is pretty much what
I wanted:
Quoted 6 lines
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
BTW, having overlapping WC, UC regions make the end result UC. So in this case, you may not be getting the desired performance.
Quoted 8 lines
So the question is - was there an easier/correct way to do this?
It might have been nice if the MTRR ioctls allowed the register number to be
specified on the Set commands, though I'm not sure that would have helped in
this case.
Quoted 12 lines · Suresh B
Quoted 10 lines · Howard Chu
So now I have this, which is pretty much
what
I wanted:reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
Quoted 2 lines
BTW, having overlapping WC, UC regions make the end result UC. So in this case, you may not be getting the desired performance.
Thanks, I noticed that later. I simply deleted the UC mapping since it was no longer needed.
Signature · 4 lines
-- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/