Skip to contentSkip to conversation

MTRR initialization

DISCUSSION Howard Chu 8 participants 16 messages Sep 14, 2007 – Sep 22, 2007

lore
Share & tools
lore

Howard Chu

author
#0 lore raw

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

The X server and various other programs try to add a mapping for my video card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this always fails with a type mismatch error (old type is write-back). Apparently it's conflicting with mapping register 0. I can't just disable the existing settings and re-add them; the system hangs soon after disabling reg01.

I guess the kernel must be getting the initial setup from the BIOS. I've hacked around this in mtrr/generic.c by explicitly changing the MTRR state in get_mtrr_state to split the first mapping into two; one at base 0 size 2048M and one at base 2048M size 1024M. So now I have this, which is pretty much what I wanted:

reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1

So the question is - was there an easier/correct way to do this?

It might have been nice if the MTRR ioctls allowed the register number to be specified on the Set commands, though I'm not sure that would have helped in this case.

Signature · 4 lines
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/
#1 lore raw
Quoted 8 lines · Howard Chu

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

BIOS should have another setup option about MTRR (continous?) then you could get

Quoted 2 lines

reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1

reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1 ===> if you have rev E...instead of rev F.

YH

author
#2 lore raw
Quoted 10 lines · Yinghai Lu
Quoted 8 lines · Howard Chu

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

Quoted 7 lines

BIOS should have another setup option about MTRR (continous?) then you could get

Quoted 2 lines

reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1

reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1 ===> if you have rev E...instead of rev F.

Thanks for the reply. Unfortunately this BIOS doesn't seem to have any other options related to the MTRRs. I guess I'll just live with this hack.

Signature · 4 lines
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/
generic.c renamed from generic.c.O +17 -0
@@ -78,6 +78,8 @@
 			base, base + step - 1, mtrr_attrib_to_str(*types));
 }
 
+static int set_mtrr_var_ranges(unsigned int index, struct mtrr_var_range *vr);
+
 /*  Grab all of the MTRR state for this CPU into *state  */
 void get_mtrr_state(void)
 {
@@ -105,6 +107,21 @@
 	mtrr_state.def_type = (lo & 0xff);
 	mtrr_state.enabled = (lo & 0xc00) >> 10;
 
+	/* Check for initial 4GB range, it should only be 3GB */
+	if (!mtrr_state.var_ranges[0].base_hi && !(mtrr_state.var_ranges[0].base_lo & 0xffffff00) &&
+		!(mtrr_state.var_ranges[0].mask_lo & 0x80000000)) {
+		/* split initial 4GB range into 2GB and 1GB */
+		for (i = num_var_ranges-1; i>0; i--)
+			mtrr_state.var_ranges[i] = mtrr_state.var_ranges[i-1];
+		mtrr_state.var_ranges[0].mask_lo |= 0x80000000;
+		mtrr_state.var_ranges[1].base_hi = 0;
+		mtrr_state.var_ranges[1].base_lo = 0x80000000 | MTRR_TYPE_WRBACK ;
+		mtrr_state.var_ranges[1].mask_hi = mtrr_state.var_ranges[2].mask_hi;
+		mtrr_state.var_ranges[1].mask_lo = mtrr_state.var_ranges[2].mask_lo;
+		for (i = 0; i < num_var_ranges; i++)
+			set_mtrr_var_ranges(i, &mtrr_state.var_ranges[i]);
+	}
+
 	if (mtrr_show) {
 		int high_width;
 
#3 lore raw
Quoted 10 lines · Howard Chu
Quoted 9 lines · Yinghai Lu
Quoted 8 lines · Howard Chu

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

and you will have problem...

So good way is get the ram top, and hole size...and reset the whole set ot var mtrr.

you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part about CONFIG_VAR_MTRR_HOLE...

but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.

YH

#4 lore raw
Quoted 23 lines · Yinghai Lu
Quoted 3 lines · Howard Chu
Quoted 2 lines · Yinghai Lu
Quoted 1 line · Howard Chu

Hi, was wondering if anyone else has been tripped up by this... I've got

4GB of

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

and you will have problem...

So good way is get the ram top, and hole size...and reset the whole set ot var mtrr.

Quoted 6 lines

you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part about CONFIG_VAR_MTRR_HOLE...

but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.

There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.

So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.

There is some difficulty there but software can always do those things safely.

Eric

author
#5 lore raw
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines

but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.

There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.

Well the BIOS is definitely doing it wrong here. As I mentioned before, it was setting up

	0-0x100000000 WB
	0xc0000000 - 0x100000000 UC
	0xc0000000 - 0xd0000000 WC

But the Intel Architecture Software Developer's Manual states that whenever any variable MTRR range overlaps with an UC MTRR range, the range remains UC. (Section 9.12.2.3). So in fact what I needed to set was

	0-2GB WB
	2-3GB WB
	3-3.25GB WC

and delete the 3-4GB UC range to get the behavior that the BIOS seems to have been intending to set up. (Relying on the default of UC for the unspecified ranges.)

Quoted 7 lines

So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.

There is some difficulty there but software can always do those things safely.

Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always UC regardless of the bits in the page table. So with the MTRR setup that the BIOS left me with, this is still a no-go. There's no way to get the desired effect without completely reinitializing the MTRRs.

Of course, this isn't the only problem with these Asus BIOSs...

Signature · 4 lines
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/
#6 lore raw
Quoted 10 lines · Howard Chu
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines

but Andi and Eric said resetting mtrr is not good... when someone from intel try to trim the MTRR for intel CPU.

There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.

Quoted 5 lines

Well the BIOS is definitely doing it wrong here. As I mentioned before, it was setting up

	0-0x100000000 WB
	0xc0000000 - 0x100000000 UC
	0xc0000000 - 0xd0000000 WC

Ok. I do agree that is wrong, and really weird to.

Quoted 9 lines

But the Intel Architecture Software Developer's Manual states that whenever any variable MTRR range overlaps with an UC MTRR range, the range remains UC. (Section 9.12.2.3). So in fact what I needed to set was

	0-2GB WB
	2-3GB WB
	3-3.25GB WC

and delete the 3-4GB UC range to get the behavior that the BIOS seems to have been intending to set up. (Relying on the default of UC for the unspecified ranges.)

Which is certainly a good way to go if your hole sizes allow and you have enough MTRRs.

Quoted 5 lines
Quoted 5 lines

So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr. This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.

Quoted 7 lines
Quoted 2 lines

There is some difficulty there but software can always do those things safely.

Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always UC regardless of the bits in the page table. So with the MTRR setup that the BIOS left me with, this is still a no-go. There's no way to get the desired effect without completely reinitializing the MTRRs.

Please look a little more closely. WC is a flavor of UC. There is a specific exception that allows page tables to promote UC to WC.

I don't seem to have the same version of this document so I can't refer you to specific sections. But I have looked it up several times and every time I have looked I have found the exception that allows PAT to promote UC to WC.

Quoted 1 line

Of course, this isn't the only problem with these Asus BIOSs...

Sure.

My point was that when we try to solve the general problem rather then a specific case there are some very practical limits on what can be done.

Now it should be mentioned that you can go in with /proc/mtrr and fix things manually on a specific machine.

Eric

#7 lore raw
In reply to Howard Chu
Quoted 5 lines · Howard Chu

So the question is - was there an easier/correct way to do this?

It might have been nice if the MTRR ioctls allowed the register number to be specified on the Set commands, though I'm not sure that would have helped in this case.

To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...

Jesse

#8 lore raw
Quoted 5 lines · Jesse Barnes

To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...

PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.

-Andi

#9 lore raw
Quoted 8 lines · Andi Kleen
Quoted 2 lines · Jesse Barnes

To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...

PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.

Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?

On my home machine for instance, the BIOS uses all 8 MTRRs leaving none for X. I hacked it by merging a couple of MTRRs but this isn't a generic solution.

Signature · 1 line
error compiling committee.c: too many arguments to function
#10 lore raw
Quoted 2 lines

Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?

That risks breaking SMM or BIOS code.

-Andi

#11 lore raw
Quoted 6 lines · Andi Kleen
Quoted 2 lines

Is there any reason not to set the MTRRs to define the entire memory as write back, and use PAT exclusively for setting cacheability?

That risks breaking SMM or BIOS code.

Yes. Too bad.

Signature · 1 line
error compiling committee.c: too many arguments to function
#12 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Jesse Barnes

To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...

PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.

Look at the original bug report. There's a bug WB region so any subsequent WC mappings via the MTRRs won't work, so you need PAT to do WC (unless you want to rewrite all your MTRR settings as Howard has to do now).

Seems like more and more machines are being setup this way, and until we have PAT users will have to jump through hoops to get good performance.

Jesse

#13 lore raw
Quoted 11 lines · Jesse Barnes
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Jesse Barnes

To do this in a nicer way (and be less vulnerable to similar BIOS funkiness) the kernel really needs full PAT support. That should allow WC over WB and WC over UC mappings to occur, at least if I'm remembering the docs right...

PAT only really helps for device driver performance optimizations. But if the basic WB MTRRs are wrong PAT cannot really salvage it.

Look at the original bug report. There's a bug WB region so any

                                              ^^^
                                          err big
#14 lore raw
In reply to Howard Chu
Quoted 27 lines · Howard Chu

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By default, my system boots up with these MTRR settings:

reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

The X server and various other programs try to add a mapping for my video card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this always
fails with a type mismatch error (old type is write-back). Apparently it's conflicting with mapping register 0. I can't just disable the existing settings
and re-add them; the system hangs soon after disabling reg01.

I guess the kernel must be getting the initial setup from the BIOS. I've hacked
around this in mtrr/generic.c by explicitly changing the MTRR state in get_mtrr_state to split the first mapping into two; one at base 0 size 2048M
and one at base 2048M size 1024M. So now I have this, which is pretty much what
I wanted:

Quoted 6 lines

reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1

BTW, having overlapping WC, UC regions make the end result UC. So in this case, you may not be getting the desired performance.

Quoted 8 lines

So the question is - was there an easier/correct way to do this?

It might have been nice if the MTRR ioctls allowed the register number to be
specified on the Set commands, though I'm not sure that would have helped in
this case.

author
#15 lore raw
Quoted 12 lines · Suresh B
Quoted 10 lines · Howard Chu

So now I have this, which is pretty much
what
I wanted:

reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1

Quoted 2 lines

BTW, having overlapping WC, UC regions make the end result UC. So in this case, you may not be getting the desired performance.

Thanks, I noticed that later. I simply deleted the UC mapping since it was no longer needed.

Signature · 4 lines
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/