Skip to contentSkip to conversation

x86: Full support of PAT

RFC PATCH In review [email protected] 5 participants 19 messages Aug 19, 2014 – Aug 25, 2014

Subsystems (6)
x86 PCI MM core Framebuffer layer +2 more subsystems
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

cover x86: Full support of PAT

The x86 architecture offers via the PAT (Page Attribute Table) a way to specify different caching modes in page table entries. The PAT MSR contains 8 entries each specifying one of 6 possible cache modes. A pte references one of those entries via 3 bits: _PAGE_PAT, _PAGE_PWT and _PAGE_PCD.

The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of _PAGE_PAT in a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.

This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.

OTOH some old processors (Pentium) don't support PAT and the Xen hypervisor has been using a different PAT MSR configuration for some time now and can't change that as this setting is part of the ABI.

This patch set abstracts the cache mode from the pte and introduces tables to translate between cache mode and pte bits (the default cache mode "write back" is hard-wired to PAT entry 0). The tables are statically initialized with values being compatible to old processors and current usage. As soon as the PAT MSR is changed (or - in case of Xen - is read at boot time) the tables are changed accordingly. Requests of mappings with special cache modes are always possible now, in case they are not supported there will be a fallback to a compatible but slower mode.

[PATCH RFC 1/3] x86: Make page cache mode a real type [PATCH RFC 2/3] x86: Enable PAT to use cache mode translation tables [PATCH RFC 3/3] Support Xen pv-domains using PAT

author maintainer
#1 lore raw

RFC PATCH 1/3 x86: Make page cache mode a real type

From: Juergen Gross <[email protected]>

At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.

This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.

To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.

---
 arch/x86/include/asm/cacheflush.h         |  34 +++++---
 arch/x86/include/asm/fb.h                 |   6 +-
 arch/x86/include/asm/io.h                 |   2 +-
 arch/x86/include/asm/pat.h                |   6 +-
 arch/x86/include/asm/pgtable.h            |  19 ++---
 arch/x86/include/asm/pgtable_types.h      |  92 +++++++++++++++++-----
 arch/x86/mm/dump_pagetables.c             |  24 +++---
 arch/x86/mm/init.c                        |  29 +++++++
 arch/x86/mm/init_64.c                     |   9 ++-
 arch/x86/mm/iomap_32.c                    |  15 ++--
 arch/x86/mm/ioremap.c                     |  63 ++++++++-------
 arch/x86/mm/pageattr.c                    |  84 ++++++++++++--------
 arch/x86/mm/pat.c                         | 126 +++++++++++++++---------------
 arch/x86/mm/pat_internal.h                |  20 ++---
 arch/x86/mm/pat_rbtree.c                  |   8 +-
 arch/x86/pci/i386.c                       |   4 +-
 drivers/video/fbdev/gbefb.c               |   3 +-
 drivers/video/fbdev/vermilion/vermilion.c |   6 +-
 18 files changed, 339 insertions(+), 211 deletions(-)
arch/x86/include/asm/cacheflush.h +21 -13
index 9863ee3..8789280 100644
@@ -9,10 +9,10 @@
 /*
  * X86 PAT uses page flags WC and Uncached together to keep track of
  * memory type of pages that have backing page struct. X86 PAT supports 3
- * different memory types, _PAGE_CACHE_WB, _PAGE_CACHE_WC and
- * _PAGE_CACHE_UC_MINUS and fourth state where page's memory type has not
+ * different memory types, _PAGE_CACHE_MODE_WB, _PAGE_CACHE_MODE_WC and
+ * _PAGE_CACHE_MODE_UC_MINUS and fourth state where page's memory type has not
  * been changed from its default (value of -1 used to denote this).
- * Note we do not support _PAGE_CACHE_UC here.
+ * Note we do not support _PAGE_CACHE_MODE_UC here.
  */
 
 #define _PGMT_DEFAULT		0
@@ -22,34 +22,36 @@
 #define _PGMT_MASK		(1UL << PG_uncached | 1UL << PG_arch_1)
 #define _PGMT_CLEAR_MASK	(~_PGMT_MASK)
 
-static inline unsigned long get_page_memtype(struct page *pg)
+static inline enum page_cache_mode get_page_memtype(struct page *pg)
 {
 	unsigned long pg_flags = pg->flags & _PGMT_MASK;
 
 	if (pg_flags == _PGMT_DEFAULT)
 		return -1;
 	else if (pg_flags == _PGMT_WC)
-		return _PAGE_CACHE_WC;
+		return _PAGE_CACHE_MODE_WC;
 	else if (pg_flags == _PGMT_UC_MINUS)
-		return _PAGE_CACHE_UC_MINUS;
+		return _PAGE_CACHE_MODE_UC_MINUS;
 	else
-		return _PAGE_CACHE_WB;
+		return _PAGE_CACHE_MODE_WB;
 }
 
-static inline void set_page_memtype(struct page *pg, unsigned long memtype)
+static inline void set_page_memtype(struct page *pg,
+				    enum page_cache_mode memtype)
 {
 	unsigned long memtype_flags = _PGMT_DEFAULT;
 	unsigned long old_flags;
 	unsigned long new_flags;
 
 	switch (memtype) {
-	case _PAGE_CACHE_WC:
+	case _PAGE_CACHE_MODE_WC:
 		memtype_flags = _PGMT_WC;
 		break;
-	case _PAGE_CACHE_UC_MINUS:
+	case _PAGE_CACHE_MODE_UC_MINUS:
 		memtype_flags = _PGMT_UC_MINUS;
 		break;
-	case _PAGE_CACHE_WB:
+	case _PAGE_CACHE_MODE_WB:
+	default:
 		memtype_flags = _PGMT_WB;
 		break;
 	}
@@ -60,8 +62,14 @@ static inline void set_page_memtype(struct page *pg, unsigned long memtype)
 	} while (cmpxchg(&pg->flags, old_flags, new_flags) != old_flags);
 }
 #else
-static inline unsigned long get_page_memtype(struct page *pg) { return -1; }
-static inline void set_page_memtype(struct page *pg, unsigned long memtype) { }
+static inline enum page_cache_mode get_page_memtype(struct page *pg)
+{
+	return -1;
+}
+static inline void set_page_memtype(struct page *pg,
+				    enum page_cache_mode memtype)
+{
+}
 #endif
 
 /*
arch/x86/include/asm/fb.h +5 -1
index 2519d06..c3766d1 100644
@@ -8,8 +8,12 @@
 static inline void fb_pgprotect(struct file *file, struct vm_area_struct *vma,
 				unsigned long off)
 {
+	unsigned long prot;
+
+	prot = pgprot_val(vma->vm_page_prot) & ~_PAGE_CACHE_MASK;
 	if (boot_cpu_data.x86 > 3)
-		pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
+		pgprot_val(vma->vm_page_prot) =
+			prot | protval_cachemode(_PAGE_CACHE_MODE_UC);
 }
 
 extern int fb_is_primary_device(struct fb_info *info);
arch/x86/include/asm/io.h +1 -1
index b8237d8..b7d9804 100644
@@ -314,7 +314,7 @@ extern void *xlate_dev_mem_ptr(unsigned long phys);
 extern void unxlate_dev_mem_ptr(unsigned long phys, void *addr);
 
 extern int ioremap_change_attr(unsigned long vaddr, unsigned long size,
-				unsigned long prot_val);
+				enum page_cache_mode pct);
 extern void __iomem *ioremap_wc(resource_size_t offset, unsigned long size);
 
 extern bool is_early_ioremap_ptep(pte_t *ptep);
arch/x86/include/asm/pat.h +3 -3
index e2c1668..65b497b 100644
@@ -13,14 +13,14 @@ static const int pat_enabled;
 extern void pat_init(void);
 
 extern int reserve_memtype(u64 start, u64 end,
-		unsigned long req_type, unsigned long *ret_type);
+		enum page_cache_mode req_pct, enum page_cache_mode *ret_pct);
 extern int free_memtype(u64 start, u64 end);
 
 extern int kernel_map_sync_memtype(u64 base, unsigned long size,
-		unsigned long flag);
+		enum page_cache_mode pct);
 
 int io_reserve_memtype(resource_size_t start, resource_size_t end,
-			unsigned long *type);
+			enum page_cache_mode *pct);
 
 void io_free_memtype(resource_size_t start, resource_size_t end);
 
arch/x86/include/asm/pgtable.h +10 -9
index 0ec0560..3d9b07e 100644
@@ -9,9 +9,10 @@
 /*
  * Macro to mark a page protection value as UC-
  */
-#define pgprot_noncached(prot)					\
-	((boot_cpu_data.x86 > 3)				\
-	 ? (__pgprot(pgprot_val(prot) | _PAGE_CACHE_UC_MINUS))	\
+#define pgprot_noncached(prot)						\
+	((boot_cpu_data.x86 > 3)					\
+	 ? (__pgprot(pgprot_val(prot) |					\
+		     protval_cachemode(_PAGE_CACHE_MODE_UC_MINUS)))	\
 	 : (prot))
 
 #ifndef __ASSEMBLY__
@@ -399,8 +400,8 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
 #define canon_pgprot(p) __pgprot(massage_pgprot(p))
 
 static inline int is_new_memtype_allowed(u64 paddr, unsigned long size,
-					 unsigned long flags,
-					 unsigned long new_flags)
+					 enum page_cache_mode pct,
+					 enum page_cache_mode new_pct)
 {
 	/*
 	 * PAT type is always WB for untracked ranges, so no need to check.
@@ -414,10 +415,10 @@ static inline int is_new_memtype_allowed(u64 paddr, unsigned long size,
 	 * - request is uncached, return cannot be write-back
 	 * - request is write-combine, return cannot be write-back
 	 */
-	if ((flags == _PAGE_CACHE_UC_MINUS &&
-	     new_flags == _PAGE_CACHE_WB) ||
-	    (flags == _PAGE_CACHE_WC &&
-	     new_flags == _PAGE_CACHE_WB)) {
+	if ((pct == _PAGE_CACHE_MODE_UC_MINUS &&
+	     new_pct == _PAGE_CACHE_MODE_WB) ||
+	    (pct == _PAGE_CACHE_MODE_WC &&
+	     new_pct == _PAGE_CACHE_MODE_WB)) {
 		return 0;
 	}
 
arch/x86/include/asm/pgtable_types.h +72 -20

130 lines not on this page · raw message

arch/x86/mm/dump_pagetables.c +11 -13
index 167ffca..52a336a 100644
@@ -122,7 +122,7 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)
 
 	if (!pgprot_val(prot)) {
 		/* Not present */
-		pt_dump_cont_printf(m, dmsg, "                          ");
+		pt_dump_cont_printf(m, dmsg, "                              ");
 	} else {
 		if (pr & _PAGE_USER)
 			pt_dump_cont_printf(m, dmsg, "USR ");
@@ -141,18 +141,16 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)
 		else
 			pt_dump_cont_printf(m, dmsg, "    ");
 
-		/* Bit 9 has a different meaning on level 3 vs 4 */
-		if (level <= 3) {
-			if (pr & _PAGE_PSE)
-				pt_dump_cont_printf(m, dmsg, "PSE ");
-			else
-				pt_dump_cont_printf(m, dmsg, "    ");
-		} else {
-			if (pr & _PAGE_PAT)
-				pt_dump_cont_printf(m, dmsg, "pat ");
-			else
-				pt_dump_cont_printf(m, dmsg, "    ");
-		}
+		/* Bit 7 has a different meaning on level 3 vs 4 */
+		if (level <= 3 && pr & _PAGE_PSE)
+			pt_dump_cont_printf(m, dmsg, "PSE ");
+		else
+			pt_dump_cont_printf(m, dmsg, "    ");
+		if ((level == 4 && pr & _PAGE_PAT) ||
+		    ((level == 3 || level == 2) && pr & _PAGE_PAT_LARGE))
+			pt_dump_cont_printf(m, dmsg, "pat ");
+		else
+			pt_dump_cont_printf(m, dmsg, "    ");
 		if (pr & _PAGE_GLOBAL)
 			pt_dump_cont_printf(m, dmsg, "GLB ");
 		else
arch/x86/mm/init.c +29 -0
index 66dba36..0500124 100644
@@ -27,6 +27,35 @@
 
 #include "mm_internal.h"
 
+/*
+ * Tables translating between page_cache_type_t and pte encoding.
+ * Minimal supported modes are defined statically, modified if more supported
+ * cache modes are available.
+ * Index into __cachemode2pte_tbl is the cachemode.
+ * Index into __pte2cachemode_tbl are the caching attribute bits of the pte
+ * (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
+ */
+uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
+	[_PAGE_CACHE_MODE_WB]		= 0,
+	[_PAGE_CACHE_MODE_WC]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_UC_MINUS]	= _PAGE_PCD,
+	[_PAGE_CACHE_MODE_UC]		= _PAGE_PCD | _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WT]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WP]		= _PAGE_PWT,
+};
+EXPORT_SYMBOL_GPL(__cachemode2pte_tbl);
+uint8_t __pte2cachemode_tbl[8] = {
+	[__pte2cm_idx(0)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD)] = _PAGE_CACHE_MODE_UC,
+	[__pte2cm_idx(_PAGE_PAT)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
+};
+EXPORT_SYMBOL_GPL(__pte2cachemode_tbl);
+
 static unsigned long __initdata pgt_buf_start;
 static unsigned long __initdata pgt_buf_end;
 static unsigned long __initdata pgt_buf_top;
arch/x86/mm/init_64.c +6 -3
index 5621c47..37ff0b5 100644
@@ -327,12 +327,15 @@ pte_t * __init populate_extra_pte(unsigned long vaddr)
  * Create large page table mappings for a range of physical addresses.
  */
 static void __init __init_extra_mapping(unsigned long phys, unsigned long size,
-						pgprot_t prot)
+					enum page_cache_mode cache)
 {
 	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
+	pgprot_t prot;
 
+	pgprot_val(prot) = pgprot_val(PAGE_KERNEL_LARGE) |
+		pgprot_val(pgprot_4k_2_large(pgprot_cachemode(cache)));
 	BUG_ON((phys & ~PMD_MASK) || (size & ~PMD_MASK));
 	for (; size; phys += PMD_SIZE, size -= PMD_SIZE) {
 		pgd = pgd_offset_k((unsigned long)__va(phys));
@@ -355,12 +358,12 @@ static void __init __init_extra_mapping(unsigned long phys, unsigned long size,
 
 void __init init_extra_mapping_wb(unsigned long phys, unsigned long size)
 {
-	__init_extra_mapping(phys, size, PAGE_KERNEL_LARGE);
+	__init_extra_mapping(phys, size, _PAGE_CACHE_MODE_WB);
 }
 
 void __init init_extra_mapping_uc(unsigned long phys, unsigned long size)
 {
-	__init_extra_mapping(phys, size, PAGE_KERNEL_LARGE_NOCACHE);
+	__init_extra_mapping(phys, size, _PAGE_CACHE_MODE_UC);
 }
 
 /*
arch/x86/mm/iomap_32.c +10 -5
index 7b179b4..91a2d3b 100644
@@ -33,17 +33,17 @@ static int is_io_mapping_possible(resource_size_t base, unsigned long size)
 
 int iomap_create_wc(resource_size_t base, unsigned long size, pgprot_t *prot)
 {
-	unsigned long flag = _PAGE_CACHE_WC;
+	enum page_cache_mode pct = _PAGE_CACHE_MODE_WC;
 	int ret;
 
 	if (!is_io_mapping_possible(base, size))
 		return -EINVAL;
 
-	ret = io_reserve_memtype(base, base + size, &flag);
+	ret = io_reserve_memtype(base, base + size, &pct);
 	if (ret)
 		return ret;
 
-	*prot = __pgprot(__PAGE_KERNEL | flag);
+	*prot = __pgprot(__PAGE_KERNEL | pgprot_val(pgprot_cachemode(pct)));
 	return 0;
 }
 EXPORT_SYMBOL_GPL(iomap_create_wc);
@@ -73,6 +73,9 @@ void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)
 /*
  * Map 'pfn' using protections 'prot'
  */
+#define __PAGE_KERNEL_WC	(__PAGE_KERNEL | \
+				 protval_cachemode(_PAGE_CACHE_TYPE_WC))
+
 void __iomem *
 iomap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)
 {
@@ -82,12 +85,14 @@ iomap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)
 	 * MTRR is UC or WC.  UC_MINUS gets the real intention, of the
 	 * user, which is "WC if the MTRR is WC, UC if you can't do that."
 	 */
-	if (!pat_enabled && pgprot_val(prot) == pgprot_val(PAGE_KERNEL_WC))
-		prot = PAGE_KERNEL_UC_MINUS;
+	if (!pat_enabled && pgprot_val(prot) == __PAGE_KERNEL_WC)
+		prot = __pgprot(__PAGE_KERNEL |
+				protval_pagemode(_PAGE_CACHE_TYPE_UC_MINUS));
 
 	return (void __force __iomem *) kmap_atomic_prot_pfn(pfn, prot);
 }
 EXPORT_SYMBOL_GPL(iomap_atomic_prot_pfn);
+#undef __PAGE_KERNEL_WC
 
 void
 iounmap_atomic(void __iomem *kvaddr)
arch/x86/mm/ioremap.c +33 -30

147 lines not on this page · raw message

arch/x86/mm/pageattr.c +50 -34

243 lines not on this page · raw message

arch/x86/mm/pat.c +65 -61

336 lines not on this page · raw message

arch/x86/mm/pat_internal.h +10 -10
index 77e5ba1..b7590d4 100644
@@ -10,30 +10,30 @@ struct memtype {
 	u64			start;
 	u64			end;
 	u64			subtree_max_end;
-	unsigned long		type;
+	enum page_cache_mode	type;
 	struct rb_node		rb;
 };
 
-static inline char *cattr_name(unsigned long flags)
+static inline char *cattr_name(enum page_cache_mode pct)
 {
-	switch (flags & _PAGE_CACHE_MASK) {
-	case _PAGE_CACHE_UC:		return "uncached";
-	case _PAGE_CACHE_UC_MINUS:	return "uncached-minus";
-	case _PAGE_CACHE_WB:		return "write-back";
-	case _PAGE_CACHE_WC:		return "write-combining";
-	default:			return "broken";
+	switch (pct) {
+	case _PAGE_CACHE_MODE_UC:		return "uncached";
+	case _PAGE_CACHE_MODE_UC_MINUS:		return "uncached-minus";
+	case _PAGE_CACHE_MODE_WB:		return "write-back";
+	case _PAGE_CACHE_MODE_WC:		return "write-combining";
+	default:				return "broken";
 	}
 }
 
 #ifdef CONFIG_X86_PAT
 extern int rbt_memtype_check_insert(struct memtype *new,
-					unsigned long *new_type);
+					enum page_cache_mode *new_type);
 extern struct memtype *rbt_memtype_erase(u64 start, u64 end);
 extern struct memtype *rbt_memtype_lookup(u64 addr);
 extern int rbt_memtype_copy_nth_element(struct memtype *out, loff_t pos);
 #else
 static inline int rbt_memtype_check_insert(struct memtype *new,
-					unsigned long *new_type)
+					enum page_cache_mode *new_type)
 { return 0; }
 static inline struct memtype *rbt_memtype_erase(u64 start, u64 end)
 { return NULL; }
arch/x86/mm/pat_rbtree.c +5 -3
index 415f6c4..6582adc 100644
@@ -122,11 +122,12 @@ static struct memtype *memtype_rb_exact_match(struct rb_root *root,
 
 static int memtype_rb_check_conflict(struct rb_root *root,
 				u64 start, u64 end,
-				unsigned long reqtype, unsigned long *newtype)
+				enum page_cache_mode reqtype,
+				enum page_cache_mode *newtype)
 {
 	struct rb_node *node;
 	struct memtype *match;
-	int found_type = reqtype;
+	enum page_cache_mode found_type = reqtype;
 
 	match = memtype_rb_lowest_match(&memtype_rbroot, start, end);
 	if (match == NULL)
@@ -187,7 +188,8 @@ static void memtype_rb_insert(struct rb_root *root, struct memtype *newdata)
 	rb_insert_augmented(&newdata->rb, root, &memtype_rb_augment_cb);
 }
 
-int rbt_memtype_check_insert(struct memtype *new, unsigned long *ret_type)
+int rbt_memtype_check_insert(struct memtype *new,
+			     enum page_cache_mode *ret_type)
 {
 	int err = 0;
 
arch/x86/pci/i386.c +2 -2
index 2ae525e..0951023 100644
@@ -433,14 +433,14 @@ int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
 		return -EINVAL;
 
 	if (pat_enabled && write_combine)
-		prot |= _PAGE_CACHE_WC;
+		prot |= protval_cachemode(_PAGE_CACHE_MODE_WC);
 	else if (pat_enabled || boot_cpu_data.x86 > 3)
 		/*
 		 * ioremap() and ioremap_nocache() defaults to UC MINUS for now.
 		 * To avoid attribute conflicts, request UC MINUS here
 		 * as well.
 		 */
-		prot |= _PAGE_CACHE_UC_MINUS;
+		prot |= protval_cachemode(_PAGE_CACHE_MODE_UC_MINUS);
 
 	prot |= _PAGE_IOMAP;	/* creating a mapping for IO */
 
drivers/video/fbdev/gbefb.c +2 -1
index 4aa56ba..703ffcd 100644
@@ -54,7 +54,8 @@ struct gbefb_par {
 #endif
 #endif
 #ifdef CONFIG_X86
-#define pgprot_fb(_prot) ((_prot) | _PAGE_PCD)
+#define pgprot_fb(_prot) (((_prot) & ~_PAGE_CACHE_MASK) |	\
+			  protval_cachemode(_PAGE_CACHE_TYPE_UC))
 #endif
 
 /*
drivers/video/fbdev/vermilion/vermilion.c +4 -2
index 048a666..121f257 100644
@@ -1004,13 +1004,15 @@ static int vmlfb_mmap(struct fb_info *info, struct vm_area_struct *vma)
 	struct vml_info *vinfo = container_of(info, struct vml_info, info);
 	unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
 	int ret;
+	unsigned long prot;
 
 	ret = vmlfb_vram_offset(vinfo, offset);
 	if (ret)
 		return -EINVAL;
 
-	pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
-	pgprot_val(vma->vm_page_prot) &= ~_PAGE_PWT;
+	prot = pgprot_val(vma->vm_page_prot) & ~_PAGE_CACHE_MASK;
+	pgprot_val(vma->vm_page_prot) =
+		prot | protval_cachemode(_PAGE_CACHE_MODE_UC);
 
 	return vm_iomap_memory(vma, vinfo->vram_start,
 			vinfo->vram_contig_size);
Signature · 1 line
1.8.4.5
#2 lore raw

Re: [PATCH RFC 1/3] x86: Make page cache mode a real type

Quoted 12 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:

From: Juergen Gross <[email protected]>

At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.

This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.

Quoted 5 lines

To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.

Hi Juergen,

Thanks for driving this. As we talked before, the changes look good to me. I will post a patchset to enable WT on top of your patchset once this is settled a bit.

I have couples of minor comments below.

arch/x86/include/asm/pgtable_types.h

Quoted 4 lines
arch/x86/include/asm/pgtable_types.h +0 -0
index f216963..7685b34 100644

:

Quoted 10 lines

/* xwr */
#define __P000 PAGE_NONE

arch/x86/include/asm/pgtable_types.h +4 -0
@@ -328,6 +331,55 @@ static inline pteval_t pte_flags(pte_t pte)
 #define pgprot_val(x)	((x).pgprot)
 #define __pgprot(x)	((pgprot_t) { (x) } )
 
+extern uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM];
+extern uint8_t __pte2cachemode_tbl[8];
+
+#define __pte2cm_idx(cb)				\

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 347

Quoted 10 lines
arch/x86/include/asm/pgtable_types.h +10 -0
+	((((cb) >> (_PAGE_BIT_PAT - 2)) & 4) |		\
+	 (((cb) >> (_PAGE_BIT_PCD - 1)) & 2) |		\
+	 (((cb) >> _PAGE_BIT_PWT) & 1))
+
+static inline unsigned long protval_cachemode(enum page_cache_mode pct)
+{
+	if (likely(pct == 0))
+		return 0;
+	return __cachemode2pte_tbl[pct];
+}

I think this function name is not intuitive. pgprot_val() works as pgprot-to-protval, but protval_cachemode() works the other way around as cachemode-to-protval.

How about renaming to cachemode_protval()?

Also, "pct" should probably be changed to "pcm".

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 351

Quoted 4 lines
arch/x86/include/asm/pgtable_types.h +4 -0
+static inline pgprot_t pgprot_cachemode(enum page_cache_mode pct)
+{
+	return __pgprot(protval_cachemode(pct));
+}

Ditto.

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 360

Quoted 9 lines
arch/x86/include/asm/pgtable_types.h +9 -0
+static inline enum page_cache_mode cachemode_pgprot(pgprot_t pgprot)
+{
+	unsigned long masked;
+
+	masked = pgprot_val(pgprot) & _PAGE_CACHE_MASK;
+	if (likely(masked == 0))
+		return 0;
+	return __pte2cachemode_tbl[__pte2cm_idx(masked)];
+}

Ditto.

Quoted 14 lines
arch/x86/mm/init.c +6 -0
index 66dba36..0500124 100644
@@ -27,6 +27,35 @@
 
 #include "mm_internal.h"
 
+/*
+ * Tables translating between page_cache_type_t and pte encoding.
+ * Minimal supported modes are defined statically, modified if more supported
+ * cache modes are available.
+ * Index into __cachemode2pte_tbl is the cachemode.
+ * Index into __pte2cachemode_tbl are the caching attribute bits of the pte

arch/x86/mm/init.c line 45

Quoted 10 lines
arch/x86/mm/init.c +10 -0
+ * (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
+ */
+uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
+	[_PAGE_CACHE_MODE_WB]		= 0,
+	[_PAGE_CACHE_MODE_WC]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_UC_MINUS]	= _PAGE_PCD,
+	[_PAGE_CACHE_MODE_UC]		= _PAGE_PCD | _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WT]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WP]		= _PAGE_PWT,
+};

I think WT and WP should be set to _PAGE_PCD (UC_MINUS) for safe.

arch/x86/mm/init.c line 56

Quoted 11 lines
arch/x86/mm/init.c +11 -0
+EXPORT_SYMBOL_GPL(__cachemode2pte_tbl);
+uint8_t __pte2cachemode_tbl[8] = {
+	[__pte2cm_idx(0)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD)] = _PAGE_CACHE_MODE_UC,
+	[__pte2cm_idx(_PAGE_PAT)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
+};

:

Quoted 12 lines
arch/x86/mm/iomap_32.c +1 -1
index 7b179b4..91a2d3b 100644
@@ -33,17 +33,17 @@ static int is_io_mapping_possible(resource_size_t base, unsigned long size)
 
 int iomap_create_wc(resource_size_t base, unsigned long size, pgprot_t *prot)
 {
-	unsigned long flag = _PAGE_CACHE_WC;
+	enum page_cache_mode pct = _PAGE_CACHE_MODE_WC;
 	int ret;
 

arch/x86/mm/iomap_32.c static int is_io_mapping_possible(resource_size_t base, unsigned long size) · line 46

Quoted 10 lines
arch/x86/mm/iomap_32.c +2 -2
 	if (!is_io_mapping_possible(base, size))
 		return -EINVAL;
 
-	ret = io_reserve_memtype(base, base + size, &flag);
+	ret = io_reserve_memtype(base, base + size, &pct);
 	if (ret)
 		return ret;
 
-	*prot = __pgprot(__PAGE_KERNEL | flag);
+	*prot = __pgprot(__PAGE_KERNEL | pgprot_val(pgprot_cachemode(pct)));

pgrot_val(pgprot_cachemode(pct)) can be simply protval_cachemode(pct). (again, this should be renamed as cachemode_protval().)

There are other places similar to this.

arch/x86/mm/iomap_32.c void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot) · line 77

Quoted 9 lines
arch/x86/mm/iomap_32.c +2 -0
 	return 0;
 }
 EXPORT_SYMBOL_GPL(iomap_create_wc);
@@ -73,6 +73,9 @@ void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)
 /*
  * Map 'pfn' using protections 'prot'
  */
+#define __PAGE_KERNEL_WC	(__PAGE_KERNEL | \
+				 protval_cachemode(_PAGE_CACHE_TYPE_WC))

_PAGE_CACHE_TYPE_WC should be _PAGE_CACHE_MODE_WC.

There are a few other places where _PAGE_CACHE_TYPE_* are still used, so please fix for that.

Thanks,
-Toshi

author maintainer
#3 lore raw

Re: [Xen-devel] [PATCH RFC 1/3] x86: Make page cache mode a real type

Quoted 57 lines · Toshi Kani
Quoted 12 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:

From: Juergen Gross <[email protected]>

At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.

This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.

Quoted 5 lines

To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.

Hi Juergen,

Thanks for driving this. As we talked before, the changes look good to me. I will post a patchset to enable WT on top of your patchset once this is settled a bit.

I have couples of minor comments below.

arch/x86/include/asm/pgtable_types.h

Quoted 4 lines
arch/x86/include/asm/pgtable_types.h +0 -0
index f216963..7685b34 100644

:

Quoted 10 lines
  /*         xwr */
  #define __P000	PAGE_NONE
arch/x86/include/asm/pgtable_types.h +4 -0
@@ -328,6 +331,55 @@ static inline pteval_t pte_flags(pte_t pte)
  #define pgprot_val(x)	((x).pgprot)
  #define __pgprot(x)	((pgprot_t) { (x) } )
 
+extern uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM];
+extern uint8_t __pte2cachemode_tbl[8];
+
+#define __pte2cm_idx(cb)				\

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 347

Quoted 10 lines
arch/x86/include/asm/pgtable_types.h +10 -0
+	((((cb) >> (_PAGE_BIT_PAT - 2)) & 4) |		\
+	 (((cb) >> (_PAGE_BIT_PCD - 1)) & 2) |		\
+	 (((cb) >> _PAGE_BIT_PWT) & 1))
+
+static inline unsigned long protval_cachemode(enum page_cache_mode pct)
+{
+	if (likely(pct == 0))
+		return 0;
+	return __cachemode2pte_tbl[pct];
+}

I think this function name is not intuitive. pgprot_val() works as pgprot-to-protval, but protval_cachemode() works the other way around as cachemode-to-protval.

Quoted 1 line

How about renaming to cachemode_protval()?

I think I'll use cachemode2protval().

Quoted 2 lines

Also, "pct" should probably be changed to "pcm".

Yeah.

Quoted 7 lines

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 351

Quoted 5 lines
arch/x86/include/asm/pgtable_types.h +4 -0
+static inline pgprot_t pgprot_cachemode(enum page_cache_mode pct)
+{
+	return __pgprot(protval_cachemode(pct));
+}

Ditto.

Will be cachemode2pgprot().

Quoted 12 lines

arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 360

Quoted 10 lines
arch/x86/include/asm/pgtable_types.h +9 -0
+static inline enum page_cache_mode cachemode_pgprot(pgprot_t pgprot)
+{
+	unsigned long masked;
+
+	masked = pgprot_val(pgprot) & _PAGE_CACHE_MASK;
+	if (likely(masked == 0))
+		return 0;
+	return __pte2cachemode_tbl[__pte2cm_idx(masked)];
+}

Ditto.

Will be pgprot2cachemode().

Quoted 19 lines
Quoted 19 lines
arch/x86/mm/init.c +10 -0
index 66dba36..0500124 100644
@@ -27,6 +27,35 @@
 
  #include "mm_internal.h"
 
+/*
+ * Tables translating between page_cache_type_t and pte encoding.
+ * Minimal supported modes are defined statically, modified if more supported
+ * cache modes are available.
+ * Index into __cachemode2pte_tbl is the cachemode.
+ * Index into __pte2cachemode_tbl are the caching attribute bits of the pte
+ * (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
+ */
+uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
+	[_PAGE_CACHE_MODE_WB]		= 0,
Quoted 8 lines

arch/x86/mm/init.c line 45

Quoted 6 lines
arch/x86/mm/init.c +6 -0
+	[_PAGE_CACHE_MODE_WC]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_UC_MINUS]	= _PAGE_PCD,
+	[_PAGE_CACHE_MODE_UC]		= _PAGE_PCD | _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WT]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WP]		= _PAGE_PWT,
+};

I think WT and WP should be set to _PAGE_PCD (UC_MINUS) for safe.

Oh, you are right.

Quoted 33 lines

arch/x86/mm/init.c line 56

Quoted 12 lines
arch/x86/mm/init.c +11 -0
+EXPORT_SYMBOL_GPL(__cachemode2pte_tbl);
+uint8_t __pte2cachemode_tbl[8] = {
+	[__pte2cm_idx(0)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD)] = _PAGE_CACHE_MODE_UC,
+	[__pte2cm_idx(_PAGE_PAT)] = _PAGE_CACHE_MODE_WB,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC,
+	[__pte2cm_idx(_PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS,
+	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
+};

:

Quoted 20 lines
arch/x86/mm/iomap_32.c +2 -2
index 7b179b4..91a2d3b 100644
@@ -33,17 +33,17 @@ static int is_io_mapping_possible(resource_size_t base, unsigned long size)
 
  int iomap_create_wc(resource_size_t base, unsigned long size, pgprot_t *prot)
  {
-	unsigned long flag = _PAGE_CACHE_WC;
+	enum page_cache_mode pct = _PAGE_CACHE_MODE_WC;
  	int ret;
 
  	if (!is_io_mapping_possible(base, size))
  		return -EINVAL;
 
-	ret = io_reserve_memtype(base, base + size, &flag);
+	ret = io_reserve_memtype(base, base + size, &pct);
  	if (ret)
  		return ret;
 
Quoted 5 lines

arch/x86/mm/iomap_32.c static int is_io_mapping_possible(resource_size_t base, unsigned long size) · line 46

Quoted 2 lines
arch/x86/mm/iomap_32.c +1 -1
-	*prot = __pgprot(__PAGE_KERNEL | flag);
+	*prot = __pgprot(__PAGE_KERNEL | pgprot_val(pgprot_cachemode(pct)));

pgrot_val(pgprot_cachemode(pct)) can be simply protval_cachemode(pct). (again, this should be renamed as cachemode_protval().)

... or cachemode2protval().

Quoted 15 lines

There are other places similar to this.

arch/x86/mm/iomap_32.c void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot) · line 77

Quoted 9 lines
arch/x86/mm/iomap_32.c +2 -0
  	return 0;
  }
  EXPORT_SYMBOL_GPL(iomap_create_wc);
@@ -73,6 +73,9 @@ void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)
  /*
   * Map 'pfn' using protections 'prot'
   */
+#define __PAGE_KERNEL_WC	(__PAGE_KERNEL | \
+				 protval_cachemode(_PAGE_CACHE_TYPE_WC))

_PAGE_CACHE_TYPE_WC should be _PAGE_CACHE_MODE_WC.

Quoted 2 lines

There are a few other places where _PAGE_CACHE_TYPE_* are still used, so please fix for that.

Oops.

Thanks, Juergen

#4 lore raw

Re: [Xen-devel] [PATCH RFC 1/3] x86: Make page cache mode a real type

Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 21.08.14 at 11:30, <[email protected]> wrote:
Quoted 17 lines · Toshi Kani
Quoted 16 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:
arch/x86/mm/init.c +10 -0
@@ -27,6 +27,35 @@
 
  #include "mm_internal.h"
 
+/*
+ * Tables translating between page_cache_type_t and pte encoding.
+ * Minimal supported modes are defined statically, modified if more supported
+ * cache modes are available.
+ * Index into __cachemode2pte_tbl is the cachemode.
+ * Index into __pte2cachemode_tbl are the caching attribute bits of the pte
+ * (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
+ */
+uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
+	[_PAGE_CACHE_MODE_WB]		= 0,
Quoted 8 lines

arch/x86/mm/init.c line 45

Quoted 6 lines
arch/x86/mm/init.c +6 -0
+	[_PAGE_CACHE_MODE_WC]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_UC_MINUS]	= _PAGE_PCD,
+	[_PAGE_CACHE_MODE_UC]		= _PAGE_PCD | _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WT]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WP]		= _PAGE_PWT,
+};

I think WT and WP should be set to _PAGE_PCD (UC_MINUS) for safe.

Quoted 1 line

Oh, you are right.

Actually I suppose the original comment was about WC and WP; defaulting WT to _PAGE_PWT seems quite correct to me.

Jan

#5 lore raw

Re: [Xen-devel] [PATCH RFC 1/3] x86: Make page cache mode a real type

Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 21.08.14 at 11:30, <[email protected]> wrote:
Quoted 25 lines · Toshi Kani
Quoted 12 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:
arch/x86/mm/init.c +6 -0
@@ -27,6 +27,35 @@
 
  #include "mm_internal.h"
 
+/*
+ * Tables translating between page_cache_type_t and pte encoding.
+ * Minimal supported modes are defined statically, modified if more supported
+ * cache modes are available.
+ * Index into __cachemode2pte_tbl is the cachemode.
+ * Index into __pte2cachemode_tbl are the caching attribute bits of the pte

arch/x86/mm/init.c line 45

Quoted 10 lines
arch/x86/mm/init.c +10 -0
+ * (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
+ */
+uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
+	[_PAGE_CACHE_MODE_WB]		= 0,
+	[_PAGE_CACHE_MODE_WC]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_UC_MINUS]	= _PAGE_PCD,
+	[_PAGE_CACHE_MODE_UC]		= _PAGE_PCD | _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WT]		= _PAGE_PWT,
+	[_PAGE_CACHE_MODE_WP]		= _PAGE_PWT,
+};

I think WT and WP should be set to _PAGE_PCD (UC_MINUS) for safe.

Quoted 4 lines
Quoted 1 line

Oh, you are right.

Actually I suppose the original comment was about WC and WP; defaulting WT to _PAGE_PWT seems quite correct to me.

My comment was about WT and WP, whose cache modes are defined here but are not supported in this patchset. _PAGE_PWT is used by WC, which has weakly ordered writes. WT and WP have strongly ordered writes, so they should be redirected to UC- for safe.

Thanks,
-Toshi

#6 lore raw

Re: [PATCH RFC 1/3] x86: Make page cache mode a real type

In reply to [email protected]
Quoted 12 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:

From: Juergen Gross <[email protected]>

At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.

This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.

Quoted 5 lines

To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.

One more comment below..

arch/x86/include/asm/cacheflush.h

Quoted 1 line
arch/x86/include/asm/cacheflush.h +0 -0

:

Quoted 13 lines
arch/x86/include/asm/cacheflush.h +3 -2
-static inline void set_page_memtype(struct page *pg, unsigned long memtype)
+static inline void set_page_memtype(struct page *pg,
+				    enum page_cache_mode memtype)
 {
 	unsigned long memtype_flags = _PGMT_DEFAULT;
 	unsigned long old_flags;
 	unsigned long new_flags;
 
 	switch (memtype) {
-	case _PAGE_CACHE_WC:
+	case _PAGE_CACHE_MODE_WC:
 		memtype_flags = _PGMT_WC;
 		break;

arch/x86/include/asm/cacheflush.h line 57

Quoted 10 lines
arch/x86/include/asm/cacheflush.h +3 -2
-	case _PAGE_CACHE_UC_MINUS:
+	case _PAGE_CACHE_MODE_UC_MINUS:
 		memtype_flags = _PGMT_UC_MINUS;
 		break;
-	case _PAGE_CACHE_WB:
+	case _PAGE_CACHE_MODE_WB:
+	default:
 		memtype_flags = _PGMT_WB;
 		break;
 	}

Adding the "default" case handled as _PGMT_WB is not correct here. free_ram_pages_type() calls set_page_memtype() with -1, which needs to be set to _PGMT_DEFAULT.

Thanks,
-Toshi

author maintainer
#7 lore raw

Re: [PATCH RFC 1/3] x86: Make page cache mode a real type

Quoted 42 lines · Toshi Kani
Quoted 12 lines · On Tue, 2014-08-19 at 15:25 +0200, [email protected] wrote:

From: Juergen Gross <[email protected]>

At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.

This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.

Quoted 5 lines

To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.

One more comment below..

arch/x86/include/asm/cacheflush.h

Quoted 1 line
arch/x86/include/asm/cacheflush.h +0 -0

:

Quoted 19 lines
arch/x86/include/asm/cacheflush.h +5 -4
-static inline void set_page_memtype(struct page *pg, unsigned long memtype)
+static inline void set_page_memtype(struct page *pg,
+				    enum page_cache_mode memtype)
  {
  	unsigned long memtype_flags = _PGMT_DEFAULT;
  	unsigned long old_flags;
  	unsigned long new_flags;
 
  	switch (memtype) {
-	case _PAGE_CACHE_WC:
+	case _PAGE_CACHE_MODE_WC:
  		memtype_flags = _PGMT_WC;
  		break;
-	case _PAGE_CACHE_UC_MINUS:
+	case _PAGE_CACHE_MODE_UC_MINUS:
  		memtype_flags = _PGMT_UC_MINUS;
  		break;
-	case _PAGE_CACHE_WB:
+	case _PAGE_CACHE_MODE_WB:
Quoted 8 lines

arch/x86/include/asm/cacheflush.h line 57

Quoted 4 lines
arch/x86/include/asm/cacheflush.h +1 -0
+	default:
  		memtype_flags = _PGMT_WB;
  		break;
  	}

Adding the "default" case handled as _PGMT_WB is not correct here. free_ram_pages_type() calls set_page_memtype() with -1, which needs to be set to _PGMT_DEFAULT.

It says so in the comment above. I'll correct it, thanks.

Juergen

author maintainer
#8 lore raw

RFC PATCH 2/3 x86: Enable PAT to use cache mode translation tables

In reply to [email protected]

From: Juergen Gross <[email protected]>

Update the translation tables from cache mode to pgprot values according to the PAT settings. This enables changing the cache attributes of a PAT index in just one place without having to change at the users side.

With this change it is possible to use the same kernel with different PAT configurations, e.g. supporting Xen.

---
 arch/x86/include/asm/pat.h           |  1 +
 arch/x86/include/asm/pgtable_types.h |  4 +++
 arch/x86/mm/init.c                   |  8 +++++
 arch/x86/mm/pat.c                    | 57 +++++++++++++++++++++++++++++++++++-
 include/linux/mm.h                   |  1 +
 5 files changed, 70 insertions(+), 1 deletion(-)
arch/x86/include/asm/pat.h +1 -0
index 65b497b..a7816e1 100644
@@ -11,6 +11,7 @@ static const int pat_enabled;
 #endif
 
 extern void pat_init(void);
+void pat_init_cache_modes(u64 pat);
 
 extern int reserve_memtype(u64 start, u64 end,
 		enum page_cache_mode req_pct, enum page_cache_mode *ret_pct);
arch/x86/include/asm/pgtable_types.h +4 -0
index 7685b34..45c720e 100644
@@ -338,6 +338,10 @@ extern uint8_t __pte2cachemode_tbl[8];
 	((((cb) >> (_PAGE_BIT_PAT - 2)) & 4) |		\
 	 (((cb) >> (_PAGE_BIT_PCD - 1)) & 2) |		\
 	 (((cb) >> _PAGE_BIT_PWT) & 1))
+#define __cm_idx2pte(i)					\
+	((((i) & 4) << (_PAGE_BIT_PAT - 2)) |		\
+	 (((i) & 2) << (_PAGE_BIT_PCD - 1)) |		\
+	 (((i) & 1) << _PAGE_BIT_PWT))
 
 static inline unsigned long protval_cachemode(enum page_cache_mode pct)
 {
arch/x86/mm/init.c +8 -0
index 0500124..d2ebe70 100644
@@ -716,3 +716,11 @@ void __init zone_sizes_init(void)
 	free_area_init_nodes(max_zone_pfns);
 }
 
+void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache)
+{
+	/* entry 0 MUST be WB (hardwired to speed up translations) */
+	BUG_ON(!entry && cache != _PAGE_CACHE_MODE_WB);
+
+	__cachemode2pte_tbl[cache] = __cm_idx2pte(entry);
+	__pte2cachemode_tbl[entry] = cache;
+}
arch/x86/mm/pat.c +56 -1

70 lines not on this page · raw message

include/linux/mm.h +1 -0
index 8981cc8..d7bf551 100644
@@ -1574,6 +1574,7 @@ extern void free_area_init(unsigned long * zones_size);
 extern void free_area_init_node(int nid, unsigned long * zones_size,
 		unsigned long zone_start_pfn, unsigned long *zholes_size);
 extern void free_initmem(void);
+extern void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache);
 
 /*
  * Free reserved pages within range [PAGE_ALIGN(start), end & PAGE_MASK)
Signature · 1 line
1.8.4.5
#9 lore raw

Re: [PATCH RFC 2/3] x86: Enable PAT to use cache mode translation tables

arch/x86/mm/pat.c void pat_init(void) · line 175

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 19.08.14 at 15:25, <[email protected]> wrote:
arch/x86/mm/pat.c +5 -1
@@ -118,8 +167,14 @@ void pat_init(void)
 	      PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, UC);
 
 	/* Boot CPU check */
-	if (!boot_pat_state)
+	if (!boot_pat_state) {
 		rdmsrl(MSR_IA32_CR_PAT, boot_pat_state);
+		/*
+		 * Init cache mode tables before writing MSR to give Xen a
+		 * chance to correct the changes when doing the write.
+		 */

This comment seems pretty odd to me: For one, a PV guest on Xen shouldn't be trying to write PAT MSR at all under the current ABI (the write will be ignored, yes, but accompanied with a warning message, which PV kernels - by the mere fact that they're PV - should try to avoid). And then "correct the changes" both gives the impression as if they were wrong and as if some of what the kernel writes may be under the kernel's control. Hence I think this code and comment should either be consistently assuming that the kernel has no control at all, or should read back the value after having written it, and set the internal tables based on the value read back.

Jan

Quoted 5 lines
arch/x86/mm/pat.c +2 -0
+		pat_init_cache_modes(pat);
+	}
 
 	wrmsrl(MSR_IA32_CR_PAT, pat);
 
#10 lore raw

Re: [Xen-devel] [PATCH RFC 2/3] x86: Enable PAT to use cache mode translation tables

In reply to [email protected]

arch/x86/mm/pat.c void pat_init(void) · line 175

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 19.08.14 at 15:25, <[email protected]> wrote:
arch/x86/mm/pat.c +5 -1
@@ -118,8 +167,14 @@ void pat_init(void)
 	      PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, UC);
 
 	/* Boot CPU check */
-	if (!boot_pat_state)
+	if (!boot_pat_state) {
 		rdmsrl(MSR_IA32_CR_PAT, boot_pat_state);
+		/*
+		 * Init cache mode tables before writing MSR to give Xen a
+		 * chance to correct the changes when doing the write.
+		 */

This comment seems pretty odd to me: For one, a PV guest on Xen shouldn't be trying to write PAT MSR at all under the current ABI (the write will be ignored, yes, but accompanied with a warning message, which PV kernels - by the mere fact that they're PV - should try to avoid). And then "correct the changes" both gives the impression as if they were wrong and as if some of what the kernel writes may be under the kernel's control. Hence I think this code and comment should either be consistently assuming that the kernel has no control at all, or should read back the value after having written it, and set the internal tables based on the value read back.

Jan

Quoted 5 lines
arch/x86/mm/pat.c +2 -0
+		pat_init_cache_modes(pat);
+	}
 
 	wrmsrl(MSR_IA32_CR_PAT, pat);
 
author maintainer
#11 lore raw

Re: [Xen-devel] [PATCH RFC 2/3] x86: Enable PAT to use cache mode translation tables

Quoted 16 lines · Jan Beulich

arch/x86/mm/pat.c void pat_init(void) · line 175

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 19.08.14 at 15:25, <[email protected]> wrote:
arch/x86/mm/pat.c +5 -1
@@ -118,8 +167,14 @@ void pat_init(void)
  	      PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, UC);
 
  	/* Boot CPU check */
-	if (!boot_pat_state)
+	if (!boot_pat_state) {
  		rdmsrl(MSR_IA32_CR_PAT, boot_pat_state);
+		/*
+		 * Init cache mode tables before writing MSR to give Xen a
+		 * chance to correct the changes when doing the write.
+		 */

This comment seems pretty odd to me: For one, a PV guest on Xen shouldn't be trying to write PAT MSR at all under the current ABI (the write will be ignored, yes, but accompanied with a warning

Quoted 8 lines

message, which PV kernels - by the mere fact that they're PV - should try to avoid). And then "correct the changes" both gives the impression as if they were wrong and as if some of what the kernel writes may be under the kernel's control. Hence I think this code and comment should either be consistently assuming that the kernel has no control at all, or should read back the value after having written it, and set the internal tables based on the value read back.

I think the latter alternative is the better one. I'll change the patch.

Juergen

author maintainer
#12 lore raw

Re: [PATCH RFC 2/3] x86: Enable PAT to use cache mode translation tables

In reply to Jan Beulich
Quoted 16 lines · Jan Beulich

arch/x86/mm/pat.c void pat_init(void) · line 175

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 19.08.14 at 15:25, <[email protected]> wrote:
arch/x86/mm/pat.c +5 -1
@@ -118,8 +167,14 @@ void pat_init(void)
  	      PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, UC);
 
  	/* Boot CPU check */
-	if (!boot_pat_state)
+	if (!boot_pat_state) {
  		rdmsrl(MSR_IA32_CR_PAT, boot_pat_state);
+		/*
+		 * Init cache mode tables before writing MSR to give Xen a
+		 * chance to correct the changes when doing the write.
+		 */

This comment seems pretty odd to me: For one, a PV guest on Xen shouldn't be trying to write PAT MSR at all under the current ABI (the write will be ignored, yes, but accompanied with a warning

Quoted 8 lines

message, which PV kernels - by the mere fact that they're PV - should try to avoid). And then "correct the changes" both gives the impression as if they were wrong and as if some of what the kernel writes may be under the kernel's control. Hence I think this code and comment should either be consistently assuming that the kernel has no control at all, or should read back the value after having written it, and set the internal tables based on the value read back.

I think the latter alternative is the better one. I'll change the patch.

Juergen

author maintainer
#13 lore raw

RFC PATCH 3/3 Support Xen pv-domains using PAT

In reply to [email protected]

From: Juergen Gross <[email protected]>

With the dynamical mapping between cache modes and pgprot values it is now possible to use all cache modes via the Xen hypervisor PAT settings in a pv domain.

All to be done is to read the PAT configuration MSR and set up the translation tables accordingly.

---
 arch/x86/xen/enlighten.c | 12 ++++++----
 arch/x86/xen/mmu.c       | 60 +++++++++++++++++-------------------------------
 arch/x86/xen/xen-ops.h   |  1 +
 3 files changed, 30 insertions(+), 43 deletions(-)
arch/x86/xen/enlighten.c +8 -4
index c0cb11f..ef705a3 100644
@@ -1552,12 +1552,16 @@ asmlinkage __visible void __init xen_start_kernel(void)
 
 	xen_init_mmu_ops();
 
+	/*
+	 * Modify the cache mode translation tables to match Xen's PAT
+	 * configuration.
+	 */
+
+	if (xen_init_cache_types())
+		__supported_pte_mask &= ~(_PAGE_PWT | _PAGE_PCD);
+
 	/* Prevent unwanted bits from being set in PTEs. */
 	__supported_pte_mask &= ~_PAGE_GLOBAL;
-#if 0
-	if (!xen_initial_domain())
-#endif
-		__supported_pte_mask &= ~(_PAGE_PWT | _PAGE_PCD);
 
 	__supported_pte_mask |= _PAGE_IOMAP;
 
arch/x86/xen/mmu.c +21 -39

81 lines not on this page · raw message

arch/x86/xen/xen-ops.h +1 -0
index 28c7e0b..da7e666 100644
@@ -34,6 +34,7 @@ extern unsigned long xen_max_p2m_pfn;
 void xen_mm_pin_all(void);
 void xen_mm_unpin_all(void);
 void xen_set_pat(u64);
+int xen_init_cache_types(void);
 
 char * __init xen_memory_setup(void);
 char * xen_auto_xlated_memory_setup(void);
Signature · 1 line
1.8.4.5
#14 lore raw
In reply to [email protected]
Quoted 7 lines

The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of _PAGE_PAT in a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.

This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.

Pentium II erratum A52 (and similar on quite a few other processors)

Problem: The Page Attribute Table (PAT) contains eight entries, which must all be initialized and considered when setting up memory types for the Pentium II processor. However, in Mode B or Mode C paging, the upper four entries do not function correctly for 4-Kbyte pages. Specifically, bit seven of page table entries that translate addresses to 4-Kbyte pages should be used as the upper bit of a three-bit index to determine the PAT entry that specifies the memory type for the page. When Mode B (CR4.PSE =
1) and/or Mode C (CR4.PAE) are enabled, the processor forces this bit to zero when determining the memory type regardless of the value in the page table entry. The upper four entries of the PAT function correctly for 2-Mbyte and 4-Mbyte large pages (specified by bit 12 of the page directory entry for those translations). Implication: Only the lower four PAT entries are useful for 4 KB translations when Mode B or C paging is used. In Mode A paging (4-Kbyte pages only), all eight entries may be used. All eight entries may be used for large pages in Mode B or C paging.

Doing this stuff for Xen also IMHO makes no sense at all. We shouldn't have a kernel full of crap to deal with Xen-isms. IFF it means the changes can also implement a mix of four PAT entries on Pentium-M or earlier CPUs with PAT errata, and the full PAT on processors without errata then IMHO it becomes a whole world more interesting.

Alan

#15 lore raw
Quoted 10 lines
Quoted 8 lines
Quoted 1 line

One Thousand Gnomes <[email protected]> 08/20/14 2:08 PM >>>

The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of _PAGE_PAT in a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.

This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.

Pentium II erratum A52 (and similar on quite a few other processors)

But you're aware that for the last two major releases Xen hasn't been supporting 32-bit CPUs at all anymore? I.e. why should we limit functionality under Xen based on errata on 64-bit-incapable CPUs?

Jan

#16 lore raw
Quoted 5 lines · Jan Beulich

But you're aware that for the last two major releases Xen hasn't been supporting 32-bit CPUs at all anymore? I.e. why should we limit functionality under Xen based on errata on 64-bit-incapable CPUs?

Because there is no fscking way we're doing this differently on 32 and 64 bits. Quite frankly the issue is if we should support this under Xen at all, but I think we can do it cleanly.

-hpa

author maintainer
#17 lore raw
In reply to One Thousand Gnomes
Quoted 29 lines · One Thousand Gnomes
Quoted 7 lines

The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of _PAGE_PAT in a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.

This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.

Pentium II erratum A52 (and similar on quite a few other processors)

Problem: The Page Attribute Table (PAT) contains eight entries, which must all be initialized and considered when setting up memory types for the Pentium II processor. However, in Mode B or Mode C paging, the upper four entries do not function correctly for 4-Kbyte pages. Specifically, bit seven of page table entries that translate addresses to 4-Kbyte pages should be used as the upper bit of a three-bit index to determine the PAT entry that specifies the memory type for the page. When Mode B (CR4.PSE =
1) and/or Mode C (CR4.PAE) are enabled, the processor forces this bit to zero when determining the memory type regardless of the value in the page table entry. The upper four entries of the PAT function correctly for 2-Mbyte and 4-Mbyte large pages (specified by bit 12 of the page directory entry for those translations). Implication: Only the lower four PAT entries are useful for 4 KB translations when Mode B or C paging is used. In Mode A paging (4-Kbyte pages only), all eight entries may be used. All eight entries may be used for large pages in Mode B or C paging.

Quoted 5 lines

Doing this stuff for Xen also IMHO makes no sense at all. We shouldn't have a kernel full of crap to deal with Xen-isms. IFF it means the changes can also implement a mix of four PAT entries on Pentium-M or earlier CPUs with PAT errata, and the full PAT on processors without errata then IMHO it becomes a whole world more interesting.

This is the case.

The patches as posted don't change PAT MSR settings at all. Using all 8 PAT MSR entries on processors without errata is easy while limit usage to only 4 entries is possible as well. It is possible to make the decision dynamically while booting the system. The only difference is the value written to the PAT MSR, as this value is used to modify the tables used for translating between cache modes and pte bits. If a driver wants to set write-through mode on a processor with the PAT erratum, the translation will result in uncached mode which is okay.

Regarding Xen: The patches don't introduce further Xen-isms. They reduce the Xen specialties by being able to use the same translation mechanisms on a native system as on a Xen based system. The only differences are the values in the translation tables, as Xen is using a different PAT configuration as native Linux (with my patches Linux could use the same PAT configuration as Xen, of course).

Juergen

#18 lore raw
In reply to One Thousand Gnomes
Quoted 2 lines · One Thousand Gnomes

Doing this stuff for Xen also IMHO makes no sense at all. We shouldn't have a kernel full of crap to deal with Xen-isms.

We already *have* a kernel full of crap to deal with Xen-isms. We screwed that pooch some 10 years ago.

Quoted 4 lines

IFF it means the changes can also implement a mix of four PAT entries on Pentium-M or earlier CPUs with PAT errata, and the full PAT on processors without errata then IMHO it becomes a whole world more interesting.

That is one of the wins, indeed, as well as handle pre-PAT processors.

-hpa