Skip to contentSkip to conversation

use MTRR for write combining if PAT is not available

RFC PATCH In review Thomas Schlichter 6 participants 11 messages Oct 10, 2009 – Oct 12, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 use MTRR for write combining if PAT is not available

Hi,

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

X.org uses libpciaccess to map the framebuffer memory. This library opens /sys/bus/pci/devices/*/resource0_wc and mmaps the memory. Unfortunately, the kernel only enables write combining if PAT is enabled, if it is not, the memory is mmapped uncached. But Xorg (respectively libpciaccess) thinks it was successfully mapped with write combining enabled and thus does not additionally set up MTRR entries.

The corresponding libpciaccess code can be found here:
http://cgit.freedesktop.org/xorg/lib/libpciaccess/tree/src/linux_sysfs.c#n501

If the kernel behavior is intentional and X.org should always set up MTRR entries, why should it use /sys/.../resource0_wc at all? I think there are 2 possibilities to make the kernel behavior consistent:

1. The mmap_wc should fail if PAT is not enabled. (libpciaccess will then map the framebuffer uncached and set up MTRR entries)
2. Use MTRR to enable write combining if PAT is not available.

In an earlier thread about ioremap_wc, Arjan van de Ven wrote that option 2 is preferred over option 1:

    http://lkml.indiana.edu/hypermail/linux/kernel/0805.3/2925.html

So, I've created the attached patch implementing option 2. For me this solves the problem with the slow Video playback due to not correctly set up MTRR entries.

Kind regards,
Thomas Schlichter

From 19fb39061825a0110d1a4feb3f83dfa3f09fb738 Mon Sep 17 00:00:00 2001
From: Thomas Schlichter <[email protected]>
Date: Thu, 8 Oct 2009 21:24:07 +0200
Subject: [PATCH] Use MTRR for write combining mmap/ioremap if PAT is not available

X.org uses libpciaccess which tries to mmap with write combining enabled via /sys/bus/pci/devices/*/resource0_wc. Currently, when PAT is not enabled, we fall back to uncached mmap. Then libpciaccess thinks it succeeded mapping with write combining anabled and does not set up suited MTRR entries. ;-(

So when falling back to uncached mapping, we better try to set up MTRR entries automatically. To match this modified PCI mmap behavior, also ioremap_wc and set_memory_wc are adjusted.

---
 arch/x86/mm/ioremap.c  |   14 +++++++++-----
 arch/x86/mm/pageattr.c |   10 ++++++++--
 arch/x86/pci/i386.c    |    6 ++++++
 3 files changed, 23 insertions(+), 7 deletions(-)
arch/x86/mm/ioremap.c +9 -5
index 334e63c..1a73bbc 100644
@@ -21,6 +21,7 @@
 #include <asm/tlbflush.h>
 #include <asm/pgalloc.h>
 #include <asm/pat.h>
+#include <asm/mtrr.h>
 
 #include "physaddr.h"
 
@@ -268,11 +269,14 @@ EXPORT_SYMBOL(ioremap_nocache);
  */
 void __iomem *ioremap_wc(resource_size_t phys_addr, unsigned long size)
 {
-	if (pat_enabled)
-		return __ioremap_caller(phys_addr, size, _PAGE_CACHE_WC,
-					__builtin_return_address(0));
-	else
-		return ioremap_nocache(phys_addr, size);
+	if (!pat_enabled) {
+		void __iomem *ret = ioremap_nocache(phys_addr, size);
+		if (ret)
+			mtrr_add(phys_addr, size, MTRR_TYPE_WRCOMB, false);
+		return ret;
+	}
+	return __ioremap_caller(phys_addr, size, _PAGE_CACHE_WC,
+				__builtin_return_address(0));
 }
 EXPORT_SYMBOL(ioremap_wc);
 
arch/x86/mm/pageattr.c +8 -2
index dd38bfb..7f3a85b 100644
@@ -23,6 +23,7 @@
 #include <asm/pgalloc.h>
 #include <asm/proto.h>
 #include <asm/pat.h>
+#include <asm/mtrr.h>
 
 /*
  * The current flushing context - we pass it instead of 5 arguments:
@@ -1010,8 +1011,13 @@ int set_memory_wc(unsigned long addr, int numpages)
 {
 	int ret;
 
-	if (!pat_enabled)
-		return set_memory_uc(addr, numpages);
+	if (!pat_enabled) {
+		ret = set_memory_uc(addr, numpages);
+		if (!ret)
+			mtrr_add(__pa(addr), numpages * PAGE_SIZE,
+				 MTRR_TYPE_WRCOMB, false);
+		return ret;
+	}
 
 	ret = reserve_memtype(__pa(addr), __pa(addr) + numpages * PAGE_SIZE,
 		_PAGE_CACHE_WC, NULL);
arch/x86/pci/i386.c +6 -0
index b22d13b..06cf678 100644
@@ -33,6 +33,7 @@
 #include <linux/bootmem.h>
 
 #include <asm/pat.h>
+#include <asm/mtrr.h>
 #include <asm/e820.h>
 #include <asm/pci_x86.h>
 #include <asm/io_apic.h>
@@ -301,5 +302,10 @@ int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
 
 	vma->vm_ops = &pci_mmap_ops;
 
+	if (!pat_enabled && write_combine)
+		mtrr_add_page(vma->vm_pgoff,
+			     (vma->vm_end - vma->vm_start) >> PAGE_SHIFT,
+			      MTRR_TYPE_WRCOMB, false);
+
 	return 0;
 }
Signature · 1 line
1.6.4.4
#1 lore raw
Quoted 3 lines · Thomas Schlichter

Hi,

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

is there any CPU left that does not support PAT ?

eg in other words, should we try instead to get PAT working for you?

#2 lore raw

Am Samstag 10 Oktober 2009 06:24:41 schrieb Arjan van de Ven:

Quoted 10 lines
Quoted 5 lines · Thomas Schlichter

Hi,

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

is there any CPU left that does not support PAT ?

eg in other words, should we try instead to get PAT working for you?

Hmm, my CPU (VIA Nano) seems to support PAT, but Ubuntu disables this feature in its kernel config. I'm using 9.04, but even 9.10 will have PAT disabled:

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-
karmic.git;a=blob;f=debian.master/config/config.common.ubuntu;h=a43cbabd02bfdf5412de6fea232db86aa5d3e742;hb=HEAD

The patch I sent does nothing if PAT is enabled, so it should be quite safe for this case. And for the case when PAT is not enabled, it simply tries to set up MTRR entries, if this fails it will still behave like it does today...

So, convincing Ubuntu to enable PAT would be a workaround, but I think the problem would still exist and should be fixed.

Kind regards,
Thomas

#3 lore raw
Quoted 3 lines · Thomas Schlichter

Hmm, my CPU (VIA Nano) seems to support PAT, but Ubuntu disables this feature in its kernel config. I'm using 9.04, but even 9.10 will have PAT disabled:

that sounds like a bad idea. But to each distro their own I suppose.

Quoted 6 lines

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-
karmic.git;a=blob;f=debian.master/config/config.common.ubuntu;h=a43cbabd02bfdf5412de6fea232db86aa5d3e742;hb=HEAD

So, convincing Ubuntu to enable PAT would be a workaround, but I think the problem would still exist and should be fixed.

I think it's time to get rid of CONFIG_PAT at all.. and just have it be there. Making 10+ year old CPU features that are essential functionality a config option... ... not a good idea to be honest.

#4 lore raw
Quoted 6 lines
Quoted 4 lines

Hmm, my CPU (VIA Nano) seems to support PAT, but Ubuntu disables this feature in its kernel config. I'm using 9.04, but even 9.10 will have PAT disabled:

that sounds like a bad idea. But to each distro their own I suppose.

Seems that Ubuntu has caught this issue, and it will probably be
resolved: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/446480

- R.

#5 lore raw
In reply to Arjan van de Ven
Quoted 13 lines · Arjan van de Ven
Quoted 3 lines · Thomas Schlichter

Hmm, my CPU (VIA Nano) seems to support PAT, but Ubuntu disables this feature in its kernel config. I'm using 9.04, but even 9.10 will have PAT disabled:

that sounds like a bad idea. But to each distro their own I suppose.

Quoted 3 lines

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-
karmic.git;a=blob;f=debian.master/config/config.common.ubuntu;h=a43cbabd02bfdf5412de6fea232db86aa5d3e742;hb=HEAD

Quoted 6 lines
Quoted 2 lines

So, convincing Ubuntu to enable PAT would be a workaround, but I think the problem would still exist and should be fixed.

I think it's time to get rid of CONFIG_PAT at all.. and just have it be there. Making 10+ year old CPU features that are essential functionality a config option... ... not a good idea to be honest.

Mind sending a patch that makes the configurability of PAT dependent on EMBEDDED or so? There's still the nopat boot option so it can still be configured by a distro if it wishes to.

Ingo

#6 lore raw
In reply to Arjan van de Ven
Quoted 7 lines · Arjan van de Ven
Quoted 2 lines · Thomas Schlichter

So, convincing Ubuntu to enable PAT would be a workaround, but I think the problem would still exist and should be fixed.

I think it's time to get rid of CONFIG_PAT at all.. and just have it be
there. Making 10+ year old CPU features that are essential functionality a config option... ... not a good idea to be honest.

I do completely agree with you in this point.

But should the current behavior of not using MTRR for write combining when PAT is disabled (for whatever reason) stay? I think it is more consistent to do whatever is necessary to enable write combining when we are asked to.

Kind regards,
Thomas

#7 lore raw
In reply to Arjan van de Ven
Quoted 5 lines · Arjan van de Ven
Quoted 3 lines

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

is there any CPU left that does not support PAT ?

A few million of them. Like every Centrino laptop out there, unless the kernel blacklist for PAT on Intel CPUs is wrong.

Quoted 1 line

eg in other words, should we try instead to get PAT working for you?

Is it even possible? Can you get Intel to fix my Pentium M? I'd be quite happy to test any microcode updates you send my way :)

As things stand right now, you can assume PAT will be there in 2015 or thereabouts. Machine models that can run Linux have a service life of around 15 years.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#8 lore raw
Quoted 8 lines · Henrique de Moraes Holschuh
Quoted 4 lines · Arjan van de Ven
Quoted 3 lines

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

is there any CPU left that does not support PAT ?

A few million of them. Like every Centrino laptop out there, unless the kernel blacklist for PAT on Intel CPUs is wrong.

afaik it is extremely conservative right now. FAR too much so.

#9 lore raw
Quoted 11 lines · Arjan van de Ven
Quoted 8 lines · Henrique de Moraes Holschuh
Quoted 4 lines · Arjan van de Ven
Quoted 3 lines

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

is there any CPU left that does not support PAT ?

A few million of them. Like every Centrino laptop out there, unless the kernel blacklist for PAT on Intel CPUs is wrong.

afaik it is extremely conservative right now. FAR too much so.

That *still* means we need the patch in this thread.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#10 lore raw
Quoted 6 lines · Henrique de Moraes Holschuh
Quoted 5 lines · Arjan van de Ven
Quoted 4 lines · Henrique de Moraes Holschuh
Quoted 3 lines · Arjan van de Ven
Quoted 3 lines

I've found a problem with X.org not setting up MTRR for the framebuffer memory. After I investigated I think this is not a X.org problem, but a kernel issue.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

is there any CPU left that does not support PAT ?

A few million of them. Like every Centrino laptop out there, unless the kernel blacklist for PAT on Intel CPUs is wrong.

afaik it is extremely conservative right now. FAR too much so.

That *still* means we need the patch in this thread.

I don't know that it is too conservative - the erratum which justifies the disablement (Core Solo/Duo erratum AE7, Pentium M erratum Y31, likely other IDs for it too) states that "A page whose PAT memory type is USWC while the relevant MTRR memory type is UC, the consolidated memory type may be treated as UC (rather than WC as specified in IA-32 Intel® Architecture Software Developer's Manual). When this erratum occurs, the memory page may be as UC (rather than WC). This may have a negative performance impact." This is exactly the case being discussed here with the framebuffer which we're trying to set as WC with existing MTRRs set (or defaulting to) UC. This not only affects Pentium M but also the original Core Solo and Core Duo CPUs which surely is millions of existing laptops. It seems pretty clear-cut and there's no workaround identified, so I think we do indeed need a patch like this one.