x86: Full support of PAT
Subsystems (5)
Juergen Gross
maintainerv3 cover x86: Full support of PAT
The x86 architecture offers via the PAT (Page Attribute Table) a way to specify different caching modes in page table entries. The PAT MSR contains 8 entries each specifying one of 6 possible cache modes. A pte references one of those entries via 3 bits: _PAGE_PAT, _PAGE_PWT and _PAGE_PCD.
The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of _PAGE_PAT in a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.
This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.
OTOH some old processors (Pentium) don't support PAT correctly and the Xen hypervisor has been using a different PAT MSR configuration for some time now and can't change that as this setting is part of the ABI.
This patch set abstracts the cache mode from the pte and introduces tables to translate between cache mode and pte bits (the default cache mode "write back" is hard-wired to PAT entry 0). The tables are statically initialized with values being compatible to old processors and current usage. As soon as the PAT MSR is changed (or - in case of Xen - is read at boot time) the tables are changed accordingly. Requests of mappings with special cache modes are always possible now, in case they are not supported there will be a fallback to a compatible but slower mode.
Summing it up, this patch set adds the following features:
- capability to support WT and WP cache modes on processors with full PAT support
- processors with no or uncorrect PAT support are still working as today, even if WT or WP cache mode are selected by drivers for some pages
- reduction of Xen special handling regarding cache mode
Changes in V3:
- corrected two minor nits (UC_MINUS, again) detected by Toshi Kani
Changes in V2:
- simplified handling of PAT MSR write under Xen as suggested by David Vrabel
- removed resetting of pat_enabled under Xen
- two small corrections requested by Toshi Kani (UC_MINUS cache mode in vermilion driver, fix 32 bit kernel build failure)
- correct build error on non-x86 arch by moving definition of update_cache_mode_entry() to x86 specific header
Changes since RFC:
- renamed functions and variables as suggested by Toshi Kani
- corrected cache mode bits for WT and WP
- modified handling of PAT MSR write under Xen as suggested by Jan Beulich
Juergen Gross (3):
x86: Make page cache mode a real type
x86: Enable PAT to use cache mode translation tables Support Xen pv-domains using PAT
arch/x86/include/asm/cacheflush.h | 38 ++++--- arch/x86/include/asm/fb.h | 6 +- arch/x86/include/asm/io.h | 2 +- arch/x86/include/asm/pat.h | 7 +- arch/x86/include/asm/pgtable.h | 19 ++-- arch/x86/include/asm/pgtable_types.h | 96 ++++++++++++---- arch/x86/mm/dump_pagetables.c | 24 ++-- arch/x86/mm/init.c | 37 ++++++ arch/x86/mm/init_64.c | 9 +- arch/x86/mm/iomap_32.c | 15 ++- arch/x86/mm/ioremap.c | 63 ++++++----- arch/x86/mm/mm_internal.h | 2 + arch/x86/mm/pageattr.c | 84 ++++++++------ arch/x86/mm/pat.c | 181 +++++++++++++++++++----------- arch/x86/mm/pat_internal.h | 22 ++-- arch/x86/mm/pat_rbtree.c | 8 +- arch/x86/pci/i386.c | 4 +- arch/x86/xen/enlighten.c | 25 ++--- arch/x86/xen/mmu.c | 48 +------- arch/x86/xen/xen-ops.h | 1 - drivers/video/fbdev/gbefb.c | 3 +- drivers/video/fbdev/vermilion/vermilion.c | 6 +- 22 files changed, 421 insertions(+), 279 deletions(-)
Signature · 1 line
1.8.4.5
Juergen Gross
maintainerPATCH v3 1/3 x86: Make page cache mode a real type
At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.
This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.
To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.
--- arch/x86/include/asm/cacheflush.h | 38 +++++---- arch/x86/include/asm/fb.h | 6 +- arch/x86/include/asm/io.h | 2 +- arch/x86/include/asm/pat.h | 6 +- arch/x86/include/asm/pgtable.h | 19 ++--- arch/x86/include/asm/pgtable_types.h | 92 +++++++++++++++++----- arch/x86/mm/dump_pagetables.c | 24 +++--- arch/x86/mm/init.c | 29 +++++++ arch/x86/mm/init_64.c | 9 ++- arch/x86/mm/iomap_32.c | 15 ++-- arch/x86/mm/ioremap.c | 63 ++++++++------- arch/x86/mm/pageattr.c | 84 ++++++++++++-------- arch/x86/mm/pat.c | 126 +++++++++++++++--------------- arch/x86/mm/pat_internal.h | 22 +++--- arch/x86/mm/pat_rbtree.c | 8 +- arch/x86/pci/i386.c | 4 +- drivers/video/fbdev/gbefb.c | 3 +- drivers/video/fbdev/vermilion/vermilion.c | 6 +- 18 files changed, 344 insertions(+), 212 deletions(-)
arch/x86/include/asm/cacheflush.h +24 -14
@@ -9,10 +9,10 @@/* * X86 PAT uses page flags WC and Uncached together to keep track of * memory type of pages that have backing page struct. X86 PAT supports 3 -* different memory types, _PAGE_CACHE_WB, _PAGE_CACHE_WC and -* _PAGE_CACHE_UC_MINUS and fourth state where page's memory type has not +* different memory types, _PAGE_CACHE_MODE_WB, _PAGE_CACHE_MODE_WC and +* _PAGE_CACHE_MODE_UC_MINUS and fourth state where page's memory type has not * been changed from its default (value of -1 used to denote this). -* Note we do not support _PAGE_CACHE_UC here. +* Note we do not support _PAGE_CACHE_MODE_UC here. */ #define _PGMT_DEFAULT 0
@@ -22,36 +22,40 @@#define _PGMT_MASK (1UL << PG_uncached | 1UL << PG_arch_1)#define _PGMT_CLEAR_MASK (~_PGMT_MASK) -static inline unsigned long get_page_memtype (struct page *pg) +static inline enum page_cache_mode get_page_memtype (struct page *pg) {unsigned long pg_flags = pg->flags & _PGMT_MASK;if (pg_flags == _PGMT_DEFAULT)return -1 ;else if (pg_flags == _PGMT_WC) -return _PAGE_CACHE_WC; +return _PAGE_CACHE_MODE_WC;else if (pg_flags == _PGMT_UC_MINUS) -return _PAGE_CACHE_UC_MINUS; +return _PAGE_CACHE_MODE_UC_MINUS;else -return _PAGE_CACHE_WB; +return _PAGE_CACHE_MODE_WB; } -static inline void set_page_memtype (struct page *pg,unsigned long memtype) +static inline void set_page_memtype (struct page *pg, +enum page_cache_mode memtype) { -unsigned long memtype_flags = _PGMT_DEFAULT; +unsigned long memtype_flags;unsigned long old_flags;unsigned long new_flags;switch (memtype) { -case _PAGE_CACHE_WC: +case _PAGE_CACHE_MODE_WC: memtype_flags = _PGMT_WC;break ; -case _PAGE_CACHE_UC_MINUS: +case _PAGE_CACHE_MODE_UC_MINUS: memtype_flags = _PGMT_UC_MINUS;break ; -case _PAGE_CACHE_WB: +case _PAGE_CACHE_MODE_WB: memtype_flags = _PGMT_WB;break ; +default : + memtype_flags = _PGMT_DEFAULT; +break ; } do {
@@ -60,8 +64,14 @@ static inline void set_page_memtype(struct page *pg, unsigned long memtype) } while (cmpxchg(&pg->flags, old_flags, new_flags) != old_flags); }#else -static inline unsigned long get_page_memtype (struct page *pg) {return -1 ; } -static inline void set_page_memtype (struct page *pg,unsigned long memtype) { } +static inline enum page_cache_mode get_page_memtype (struct page *pg) +{ +return -1 ; +} +static inline void set_page_memtype (struct page *pg, +enum page_cache_mode memtype) +{ +}#endif /*
arch/x86/include/asm/fb.h +5 -1
@@ -8,8 +8,12 @@static inline void fb_pgprotect (struct file *file,struct vm_area_struct *vma,unsigned long off) { +unsigned long prot; + + prot =pgprot_val (vma->vm_page_prot) & ~_PAGE_CACHE_MASK;if (boot_cpu_data.x86 >3 ) -pgprot_val (vma->vm_page_prot) |= _PAGE_PCD; +pgprot_val (vma->vm_page_prot) = + prot |cachemode2protval (_PAGE_CACHE_MODE_UC_MINUS); }extern int fb_is_primary_device (struct fb_info *info);
arch/x86/include/asm/io.h +1 -1
@@ -314,7 +314,7 @@ extern void *xlate_dev_mem_ptr(unsigned long phys);extern void unxlate_dev_mem_ptr (unsigned long phys,void *addr);extern int ioremap_change_attr (unsigned long vaddr,unsigned long size, -unsigned long prot_val); +enum page_cache_mode pcm);extern void __iomem *ioremap_wc (resource_size_t offset,unsigned long size);extern bool is_early_ioremap_ptep (pte_t *ptep);
arch/x86/include/asm/pat.h +3 -3
@@ -13,14 +13,14 @@ static const int pat_enabled;extern void pat_init (void );extern int reserve_memtype (u64 start,u64 end, -unsigned long req_type,unsigned long *ret_type); +enum page_cache_mode req_pcm,enum page_cache_mode *ret_pcm);extern int free_memtype (u64 start,u64 end);extern int kernel_map_sync_memtype (u64 base,unsigned long size, -unsigned long flag); +enum page_cache_mode pcm);int io_reserve_memtype (resource_size_t start,resource_size_t end, -unsigned long *type); +enum page_cache_mode *pcm);void io_free_memtype (resource_size_t start,resource_size_t end);
arch/x86/include/asm/pgtable.h +10 -9
@@ -9,9 +9,10 @@/* * Macro to mark a page protection value as UC- */ -#define pgprot_noncached (prot) \ - ((boot_cpu_data.x86 > 3) \ - ? (__pgprot(pgprot_val(prot) | _PAGE_CACHE_UC_MINUS)) \ +#define pgprot_noncached (prot) \ + ((boot_cpu_data.x86 > 3) \ + ? (__pgprot(pgprot_val(prot) | \ + cachemode2protval(_PAGE_CACHE_MODE_UC_MINUS))) \ : (prot))#ifndef __ASSEMBLY__
@@ -404,8 +405,8 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)#define canon_pgprot (p) __pgprot(massage_pgprot(p))static inline int is_new_memtype_allowed (u64 paddr,unsigned long size, -unsigned long flags, -unsigned long new_flags) +enum page_cache_mode pcm, +enum page_cache_mode new_pcm) {/* * PAT type is always WB for untracked ranges, so no need to check.
@@ -419,10 +420,10 @@ static inline int is_new_memtype_allowed(u64 paddr, unsigned long size,* - request is uncached, return cannot be write-back * - request is write-combine, return cannot be write-back */ -if ((flags == _PAGE_CACHE_UC_MINUS && - new_flags == _PAGE_CACHE_WB) || - (flags == _PAGE_CACHE_WC && - new_flags == _PAGE_CACHE_WB)) { +if ((pcm == _PAGE_CACHE_MODE_UC_MINUS && + new_pcm == _PAGE_CACHE_MODE_WB) || + (pcm == _PAGE_CACHE_MODE_WC && + new_pcm == _PAGE_CACHE_MODE_WB)) {return 0 ; }
arch/x86/include/asm/pgtable_types.h +72 -20
130 lines not on this page · raw message
arch/x86/mm/dump_pagetables.c +11 -13
@@ -122,7 +122,7 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)if (!pgprot_val (prot)) {/* Not present */ -pt_dump_cont_printf (m, dmsg," " ); +pt_dump_cont_printf (m, dmsg," " ); }else {if (pr & _PAGE_USER)pt_dump_cont_printf (m, dmsg,"USR " );
@@ -141,18 +141,16 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)else pt_dump_cont_printf (m, dmsg," " ); -/* Bit 9 has a different meaning on level 3 vs 4 */ -if (level <=3 ) { -if (pr & _PAGE_PSE) -pt_dump_cont_printf (m, dmsg,"PSE " ); -else -pt_dump_cont_printf (m, dmsg," " ); - }else { -if (pr & _PAGE_PAT) -pt_dump_cont_printf (m, dmsg,"pat " ); -else -pt_dump_cont_printf (m, dmsg," " ); - } +/* Bit 7 has a different meaning on level 3 vs 4 */ +if (level <=3 && pr & _PAGE_PSE) +pt_dump_cont_printf (m, dmsg,"PSE " ); +else +pt_dump_cont_printf (m, dmsg," " ); +if ((level ==4 && pr & _PAGE_PAT) || + ((level ==3 || level ==2 ) && pr & _PAGE_PAT_LARGE)) +pt_dump_cont_printf (m, dmsg,"pat " ); +else +pt_dump_cont_printf (m, dmsg," " );if (pr & _PAGE_GLOBAL)pt_dump_cont_printf (m, dmsg,"GLB " );else
arch/x86/mm/init.c +29 -0
@@ -27,6 +27,35 @@#include "mm_internal.h" +/* +* Tables translating between page_cache_type_t and pte encoding. +* Minimal supported modes are defined statically, modified if more supported +* cache modes are available. +* Index into __cachemode2pte_tbl is the cachemode. +* Index into __pte2cachemode_tbl are the caching attribute bits of the pte +* (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2. +*/ +uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = { + [_PAGE_CACHE_MODE_WB] =0 , + [_PAGE_CACHE_MODE_WC] = _PAGE_PWT, + [_PAGE_CACHE_MODE_UC_MINUS] = _PAGE_PCD, + [_PAGE_CACHE_MODE_UC] = _PAGE_PCD | _PAGE_PWT, + [_PAGE_CACHE_MODE_WT] = _PAGE_PCD, + [_PAGE_CACHE_MODE_WP] = _PAGE_PCD, +}; +EXPORT_SYMBOL_GPL (__cachemode2pte_tbl); +uint8_t __pte2cachemode_tbl[8 ] = { + [__pte2cm_idx (0 )] = _PAGE_CACHE_MODE_WB, + [__pte2cm_idx (_PAGE_PWT)] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx (_PAGE_PCD)] = _PAGE_CACHE_MODE_UC_MINUS, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PCD)] = _PAGE_CACHE_MODE_UC, + [__pte2cm_idx (_PAGE_PAT)] = _PAGE_CACHE_MODE_WB, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx (_PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC, +}; +EXPORT_SYMBOL_GPL (__pte2cachemode_tbl); +static unsigned long __initdata pgt_buf_start;static unsigned long __initdata pgt_buf_end;static unsigned long __initdata pgt_buf_top;
arch/x86/mm/init_64.c +6 -3
@@ -327,12 +327,15 @@ pte_t * __init populate_extra_pte(unsigned long vaddr)* Create large page table mappings for a range of physical addresses. */ static void __init __init_extra_mapping (unsigned long phys,unsigned long size, -pgprot_t prot) +enum page_cache_mode cache) {pgd_t *pgd;pud_t *pud;pmd_t *pmd; +pgprot_t prot; +pgprot_val (prot) =pgprot_val (PAGE_KERNEL_LARGE ) | +pgprot_val (pgprot_4k_2_large (cachemode2pgprot (cache)));BUG_ON ((phys & ~PMD_MASK ) || (size & ~PMD_MASK ));for (; size; phys +=PMD_SIZE , size -=PMD_SIZE ) { pgd =pgd_offset_k ((unsigned long )__va (phys));
@@ -355,12 +358,12 @@ static void __init __init_extra_mapping(unsigned long phys, unsigned long size,void __init init_extra_mapping_wb (unsigned long phys,unsigned long size) { -__init_extra_mapping (phys, size,PAGE_KERNEL_LARGE ); +__init_extra_mapping (phys, size, _PAGE_CACHE_MODE_WB); }void __init init_extra_mapping_uc (unsigned long phys,unsigned long size) { -__init_extra_mapping (phys, size,PAGE_KERNEL_LARGE_NOCACHE ); +__init_extra_mapping (phys, size, _PAGE_CACHE_MODE_UC); }/*
arch/x86/mm/iomap_32.c +10 -5
@@ -33,17 +33,17 @@ static int is_io_mapping_possible(resource_size_t base, unsigned long size)int iomap_create_wc (resource_size_t base,unsigned long size,pgprot_t *prot) { -unsigned long flag = _PAGE_CACHE_WC; +enum page_cache_mode pcm = _PAGE_CACHE_MODE_WC;int ret;if (!is_io_mapping_possible (base, size))return -EINVAL ; - ret =io_reserve_memtype (base, base + size, &flag); + ret =io_reserve_memtype (base, base + size, &pcm);if (ret)return ret; - *prot =__pgprot (__PAGE_KERNEL | flag); + *prot =__pgprot (__PAGE_KERNEL |cachemode2protval (pcm));return 0 ; }EXPORT_SYMBOL_GPL (iomap_create_wc);
@@ -73,6 +73,9 @@ void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)/* * Map 'pfn' using protections 'prot' */ +#define __PAGE_KERNEL_WC (__PAGE_KERNEL | \ + cachemode2protval(_PAGE_CACHE_MODE_WC)) +void __iomem *iomap_atomic_prot_pfn (unsigned long pfn,pgprot_t prot) {
@@ -82,12 +85,14 @@ iomap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot)* MTRR is UC or WC. UC_MINUS gets the real intention, of the * user, which is "WC if the MTRR is WC, UC if you can't do that." */ -if (!pat_enabled &&pgprot_val (prot) ==pgprot_val (PAGE_KERNEL_WC )) - prot =PAGE_KERNEL_UC_MINUS ; +if (!pat_enabled &&pgprot_val (prot) == __PAGE_KERNEL_WC) + prot =__pgprot (__PAGE_KERNEL | +cachemode2protval (_PAGE_CACHE_MODE_UC_MINUS));return (void __force __iomem *)kmap_atomic_prot_pfn (pfn, prot); }EXPORT_SYMBOL_GPL (iomap_atomic_prot_pfn); +#undef __PAGE_KERNEL_WCvoid iounmap_atomic (void __iomem *kvaddr)
arch/x86/mm/ioremap.c +33 -30
147 lines not on this page · raw message
arch/x86/mm/pageattr.c +50 -34
243 lines not on this page · raw message
arch/x86/mm/pat.c +65 -61
336 lines not on this page · raw message
arch/x86/mm/pat_internal.h +12 -10
@@ -10,30 +10,32 @@ struct memtype {u64 start;u64 end;u64 subtree_max_end; -unsigned long type; +enum page_cache_mode type;struct rb_node rb; }; -static inline char *cattr_name (unsigned long flags) +static inline char *cattr_name (enum page_cache_mode pcm) { -switch (flags & _PAGE_CACHE_MASK) { -case _PAGE_CACHE_UC:return "uncached" ; -case _PAGE_CACHE_UC_MINUS:return "uncached-minus" ; -case _PAGE_CACHE_WB:return "write-back" ; -case _PAGE_CACHE_WC:return "write-combining" ; -default :return "broken" ; +switch (pcm) { +case _PAGE_CACHE_MODE_UC:return "uncached" ; +case _PAGE_CACHE_MODE_UC_MINUS:return "uncached-minus" ; +case _PAGE_CACHE_MODE_WB:return "write-back" ; +case _PAGE_CACHE_MODE_WC:return "write-combining" ; +case _PAGE_CACHE_MODE_WT:return "write-through" ; +case _PAGE_CACHE_MODE_WP:return "write-protected" ; +default :return "broken" ; } }#ifdef CONFIG_X86_PAT extern int rbt_memtype_check_insert (struct memtype *new, -unsigned long *new_type); +enum page_cache_mode *new_type);extern struct memtype *rbt_memtype_erase (u64 start,u64 end);extern struct memtype *rbt_memtype_lookup (u64 addr);extern int rbt_memtype_copy_nth_element (struct memtype *out,loff_t pos);#else static inline int rbt_memtype_check_insert (struct memtype *new, -unsigned long *new_type) +enum page_cache_mode *new_type) {return 0 ; }static inline struct memtype *rbt_memtype_erase (u64 start,u64 end) {return NULL ; }
arch/x86/mm/pat_rbtree.c +5 -3
@@ -122,11 +122,12 @@ static struct memtype *memtype_rb_exact_match(struct rb_root *root,static int memtype_rb_check_conflict (struct rb_root *root,u64 start,u64 end, -unsigned long reqtype,unsigned long *newtype) +enum page_cache_mode reqtype, +enum page_cache_mode *newtype) {struct rb_node *node;struct memtype *match; -int found_type = reqtype; +enum page_cache_mode found_type = reqtype; match =memtype_rb_lowest_match (&memtype_rbroot, start, end);if (match ==NULL )
@@ -187,7 +188,8 @@ static void memtype_rb_insert(struct rb_root *root, struct memtype *newdata)rb_insert_augmented (&newdata->rb, root, &memtype_rb_augment_cb); } -int rbt_memtype_check_insert (struct memtype *new,unsigned long *ret_type) +int rbt_memtype_check_insert (struct memtype *new, +enum page_cache_mode *ret_type) {int err =0 ;
arch/x86/pci/i386.c +2 -2
@@ -433,14 +433,14 @@ int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,return -EINVAL ;if (pat_enabled && write_combine) - prot |= _PAGE_CACHE_WC; + prot |=cachemode2protval (_PAGE_CACHE_MODE_WC);else if (pat_enabled || boot_cpu_data.x86 >3 )/* * ioremap() and ioremap_nocache() defaults to UC MINUS for now. * To avoid attribute conflicts, request UC MINUS here * as well. */ - prot |= _PAGE_CACHE_UC_MINUS; + prot |=cachemode2protval (_PAGE_CACHE_MODE_UC_MINUS); prot |= _PAGE_IOMAP;/* creating a mapping for IO */
drivers/video/fbdev/gbefb.c +2 -1
@@ -54,7 +54,8 @@ struct gbefb_par {#endif #endif #ifdef CONFIG_X86 -#define pgprot_fb (_prot) ((_prot) | _PAGE_PCD) +#define pgprot_fb (_prot) (((_prot) & ~_PAGE_CACHE_MASK) | \ + cachemode2protval(_PAGE_CACHE_MODE_UC_MINUS))#endif /*
drivers/video/fbdev/vermilion/vermilion.c +4 -2
@@ -1004,13 +1004,15 @@ static int vmlfb_mmap(struct fb_info *info, struct vm_area_struct *vma) struct vml_info *vinfo = container_of(info, struct vml_info, info);unsigned long offset = vma->vm_pgoff <<PAGE_SHIFT ;int ret; +unsigned long prot; ret =vmlfb_vram_offset (vinfo, offset);if (ret)return -EINVAL ; -pgprot_val (vma->vm_page_prot) |= _PAGE_PCD; -pgprot_val (vma->vm_page_prot) &= ~_PAGE_PWT; + prot =pgprot_val (vma->vm_page_prot) & ~_PAGE_CACHE_MASK; +pgprot_val (vma->vm_page_prot) = + prot |cachemode2protval (_PAGE_CACHE_MODE_UC_MINUS);return vm_iomap_memory (vma, vinfo->vram_start, vinfo->vram_contig_size);
Signature · 1 line
1.8.4.5
Re: [PATCH V3 1/3] x86: Make page cache mode a real type
Quoted 17 lines · Juergen Gross
At the moment there are a lot of places that handle setting or getting the page cache mode by treating the pgprot bits equal to the cache mode. This is only true because there are a lot of assumptions about the setup of the PAT MSR. Otherwise the cache type needs to get translated into pgprot bits and vice versa.
This patch tries to prepare for that by introducing a seperate type for the cache mode and adding functions to translate between those and pgprot values.
To avoid too much performance penalty the translation between cache mode and pgprot values is done via tables which contain the relevant information. Write-back cache mode is hard-wired to be 0, all other modes are configurable via those tables. For large pages there are translation functions as the PAT bit is located at different positions in the ptes of 4k and large pages.
Quoted 3 lines
Signed-off-by Stefan Bader <[email protected]>Signed-off-by Juergen Gross <[email protected]>Reviewed-by Toshi Kani <[email protected]>
with some comments below.
Quoted 285 lines
--- arch/x86/include/asm/cacheflush.h | 38 +++++---- arch/x86/include/asm/fb.h | 6 +- arch/x86/include/asm/io.h | 2 +- arch/x86/include/asm/pat.h | 6 +- arch/x86/include/asm/pgtable.h | 19 ++--- arch/x86/include/asm/pgtable_types.h | 92 +++++++++++++++++----- arch/x86/mm/dump_pagetables.c | 24 +++--- arch/x86/mm/init.c | 29 +++++++ arch/x86/mm/init_64.c | 9 ++- arch/x86/mm/iomap_32.c | 15 ++-- arch/x86/mm/ioremap.c | 63 ++++++++------- arch/x86/mm/pageattr.c | 84 ++++++++++++-------- arch/x86/mm/pat.c | 126 +++++++++++++++--------------- arch/x86/mm/pat_internal.h | 22 +++--- arch/x86/mm/pat_rbtree.c | 8 +- arch/x86/pci/i386.c | 4 +- drivers/video/fbdev/gbefb.c | 3 +- drivers/video/fbdev/vermilion/vermilion.c | 6 +- 18 files changed, 344 insertions(+), 212 deletions(-)arch/x86/include/asm/cacheflush.h +24
-14@@ -9,10 +9,10 @@/* * X86 PAT uses page flags WC and Uncached together to keep track of * memory type of pages that have backing page struct. X86 PAT supports 3 -* different memory types, _PAGE_CACHE_WB, _PAGE_CACHE_WC and -* _PAGE_CACHE_UC_MINUS and fourth state where page's memory type has not +* different memory types, _PAGE_CACHE_MODE_WB, _PAGE_CACHE_MODE_WC and +* _PAGE_CACHE_MODE_UC_MINUS and fourth state where page's memory type has not * been changed from its default (value of -1 used to denote this). -* Note we do not support _PAGE_CACHE_UC here. +* Note we do not support _PAGE_CACHE_MODE_UC here. */ #define _PGMT_DEFAULT 0@@ -22,36 +22,40 @@#define _PGMT_MASK (1UL << PG_uncached | 1UL << PG_arch_1)#define _PGMT_CLEAR_MASK (~_PGMT_MASK) -static inline unsigned long get_page_memtype (struct page *pg) +static inline enum page_cache_mode get_page_memtype (struct page *pg) {unsigned long pg_flags = pg->flags & _PGMT_MASK;if (pg_flags == _PGMT_DEFAULT)return -1 ;else if (pg_flags == _PGMT_WC) -return _PAGE_CACHE_WC; +return _PAGE_CACHE_MODE_WC;else if (pg_flags == _PGMT_UC_MINUS) -return _PAGE_CACHE_UC_MINUS; +return _PAGE_CACHE_MODE_UC_MINUS;else -return _PAGE_CACHE_WB; +return _PAGE_CACHE_MODE_WB; } -static inline void set_page_memtype (struct page *pg,unsigned long memtype) +static inline void set_page_memtype (struct page *pg, +enum page_cache_mode memtype) { -unsigned long memtype_flags = _PGMT_DEFAULT; +unsigned long memtype_flags;unsigned long old_flags;unsigned long new_flags;switch (memtype) { -case _PAGE_CACHE_WC: +case _PAGE_CACHE_MODE_WC: memtype_flags = _PGMT_WC;break ; -case _PAGE_CACHE_UC_MINUS: +case _PAGE_CACHE_MODE_UC_MINUS: memtype_flags = _PGMT_UC_MINUS;break ; -case _PAGE_CACHE_WB: +case _PAGE_CACHE_MODE_WB: memtype_flags = _PGMT_WB;break ; +default : + memtype_flags = _PGMT_DEFAULT; +break ; } do {@@ -60,8 +64,14 @@ static inline void set_page_memtype(struct page *pg, unsigned long memtype) } while (cmpxchg(&pg->flags, old_flags, new_flags) != old_flags); }#else -static inline unsigned long get_page_memtype (struct page *pg) {return -1 ; } -static inline void set_page_memtype (struct page *pg,unsigned long memtype) { } +static inline enum page_cache_mode get_page_memtype (struct page *pg) +{ +return -1 ; +} +static inline void set_page_memtype (struct page *pg, +enum page_cache_mode memtype) +{ +}#endif /* arch/x86/include/asm/fb.h +5
-1@@ -8,8 +8,12 @@static inline void fb_pgprotect (struct file *file,struct vm_area_struct *vma,unsigned long off) { +unsigned long prot; + + prot =pgprot_val (vma->vm_page_prot) & ~_PAGE_CACHE_MASK;if (boot_cpu_data.x86 >3 ) -pgprot_val (vma->vm_page_prot) |= _PAGE_PCD; +pgprot_val (vma->vm_page_prot) = + prot |cachemode2protval (_PAGE_CACHE_MODE_UC_MINUS); }extern int fb_is_primary_device (struct fb_info *info);arch/x86/include/asm/io.h +1
-1@@ -314,7 +314,7 @@ extern void *xlate_dev_mem_ptr(unsigned long phys);extern void unxlate_dev_mem_ptr (unsigned long phys,void *addr);extern int ioremap_change_attr (unsigned long vaddr,unsigned long size, -unsigned long prot_val); +enum page_cache_mode pcm);extern void __iomem *ioremap_wc (resource_size_t offset,unsigned long size);extern bool is_early_ioremap_ptep (pte_t *ptep);arch/x86/include/asm/pat.h +3
-3@@ -13,14 +13,14 @@ static const int pat_enabled;extern void pat_init (void );extern int reserve_memtype (u64 start,u64 end, -unsigned long req_type,unsigned long *ret_type); +enum page_cache_mode req_pcm,enum page_cache_mode *ret_pcm);extern int free_memtype (u64 start,u64 end);extern int kernel_map_sync_memtype (u64 base,unsigned long size, -unsigned long flag); +enum page_cache_mode pcm);int io_reserve_memtype (resource_size_t start,resource_size_t end, -unsigned long *type); +enum page_cache_mode *pcm);void io_free_memtype (resource_size_t start,resource_size_t end);arch/x86/include/asm/pgtable.h +10
-9@@ -9,9 +9,10 @@/* * Macro to mark a page protection value as UC- */ -#define pgprot_noncached (prot) \ - ((boot_cpu_data.x86 > 3) \ - ? (__pgprot(pgprot_val(prot) | _PAGE_CACHE_UC_MINUS)) \ +#define pgprot_noncached (prot) \ + ((boot_cpu_data.x86 > 3) \ + ? (__pgprot(pgprot_val(prot) | \ + cachemode2protval(_PAGE_CACHE_MODE_UC_MINUS))) \ : (prot))#ifndef __ASSEMBLY__@@ -404,8 +405,8 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)#define canon_pgprot (p) __pgprot(massage_pgprot(p))static inline int is_new_memtype_allowed (u64 paddr,unsigned long size, -unsigned long flags, -unsigned long new_flags) +enum page_cache_mode pcm, +enum page_cache_mode new_pcm) {/* * PAT type is always WB for untracked ranges, so no need to check. @@ -419,10 +420,10 @@ static inline int is_new_memtype_allowed(u64 paddr, unsigned long size,* - request is uncached, return cannot be write-back * - request is write-combine, return cannot be write-back */ -if ((flags == _PAGE_CACHE_UC_MINUS && - new_flags == _PAGE_CACHE_WB) || - (flags == _PAGE_CACHE_WC && - new_flags == _PAGE_CACHE_WB)) { +if ((pcm == _PAGE_CACHE_MODE_UC_MINUS && + new_pcm == _PAGE_CACHE_MODE_WB) || + (pcm == _PAGE_CACHE_MODE_WC && + new_pcm == _PAGE_CACHE_MODE_WB)) {return 0 ; }arch/x86/include/asm/pgtable_types.h +23
-20@@ -129,11 +129,28 @@ _PAGE_SOFT_DIRTY | _PAGE_NUMA)#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE | _PAGE_NUMA) -#define _PAGE_CACHE_MASK (_PAGE_PCD | _PAGE_PWT) -#define _PAGE_CACHE_WB (0) -#define _PAGE_CACHE_WC (_PAGE_PWT) -#define _PAGE_CACHE_UC_MINUS (_PAGE_PCD) -#define _PAGE_CACHE_UC (_PAGE_PCD | _PAGE_PWT) +/* +* The cache modes defined here are used to translate between pure SW usage +* and the HW defined cache mode bits and/or PAT entries. +* +* The resulting bits for PWT, PCD and PAT should be chosen in a way +* to have the WB mode at index 0 (all bits clear). This is the default +* right now and likely would break too much if changed. +*/ +#ifndef __ASSEMBLY__ +enum page_cache_mode { + _PAGE_CACHE_MODE_WB =0 , + _PAGE_CACHE_MODE_WC =1 , + _PAGE_CACHE_MODE_UC_MINUS =2 , + _PAGE_CACHE_MODE_UC =3 , + _PAGE_CACHE_MODE_WT =4 , + _PAGE_CACHE_MODE_WP =5 , + _PAGE_CACHE_MODE_NUM =8 +}; +#endif + +#define _PAGE_CACHE_MASK (_PAGE_PAT | _PAGE_PCD | _PAGE_PWT) +#define _PAGE_NOCACHE (cachemode2protval(_PAGE_CACHE_MODE_UC))#define PAGE_NONE __pgprot(_PAGE_PROTNONE | _PAGE_ACCESSED)#define PAGE_SHARED __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | \@@ -157,41 +174,27 @@#define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW)#define __PAGE_KERNEL_RX (__PAGE_KERNEL_EXEC & ~_PAGE_RW) -#define __PAGE_KERNEL_EXEC_NOCACHE (__PAGE_KERNEL_EXEC | _PAGE_PCD | _PAGE_PWT) -#define __PAGE_KERNEL_WC (__PAGE_KERNEL | _PAGE_CACHE_WC) -#define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_PCD | _PAGE_PWT) -#define __PAGE_KERNEL_UC_MINUS (__PAGE_KERNEL | _PAGE_PCD) +#define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_NOCACHE)#define __PAGE_KERNEL_VSYSCALL (__PAGE_KERNEL_RX | _PAGE_USER)#define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) -#define __PAGE_KERNEL_VVAR_NOCACHE (__PAGE_KERNEL_VVAR | _PAGE_PCD | _PAGE_PWT)#define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) -#define __PAGE_KERNEL_LARGE_NOCACHE (__PAGE_KERNEL | _PAGE_CACHE_UC | _PAGE_PSE)#define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE)#define __PAGE_KERNEL_IO (__PAGE_KERNEL | _PAGE_IOMAP)#define __PAGE_KERNEL_IO_NOCACHE (__PAGE_KERNEL_NOCACHE | _PAGE_IOMAP) -#define __PAGE_KERNEL_IO_UC_MINUS (__PAGE_KERNEL_UC_MINUS | _PAGE_IOMAP) -#define __PAGE_KERNEL_IO_WC (__PAGE_KERNEL_WC | _PAGE_IOMAP)#define PAGE_KERNEL __pgprot(__PAGE_KERNEL)#define PAGE_KERNEL_RO __pgprot(__PAGE_KERNEL_RO)#define PAGE_KERNEL_EXEC __pgprot(__PAGE_KERNEL_EXEC)#define PAGE_KERNEL_RX __pgprot(__PAGE_KERNEL_RX) -#define PAGE_KERNEL_WC __pgprot(__PAGE_KERNEL_WC)#define PAGE_KERNEL_NOCACHE __pgprot(__PAGE_KERNEL_NOCACHE) -#define PAGE_KERNEL_UC_MINUS __pgprot(__PAGE_KERNEL_UC_MINUS) -#define PAGE_KERNEL_EXEC_NOCACHE __pgprot(__PAGE_KERNEL_EXEC_NOCACHE)#define PAGE_KERNEL_LARGE __pgprot(__PAGE_KERNEL_LARGE) -#define PAGE_KERNEL_LARGE_NOCACHE __pgprot(__PAGE_KERNEL_LARGE_NOCACHE)#define PAGE_KERNEL_LARGE_EXEC __pgprot(__PAGE_KERNEL_LARGE_EXEC)#define PAGE_KERNEL_VSYSCALL __pgprot(__PAGE_KERNEL_VSYSCALL)#define PAGE_KERNEL_VVAR __pgprot(__PAGE_KERNEL_VVAR) -#define PAGE_KERNEL_VVAR_NOCACHE __pgprot(__PAGE_KERNEL_VVAR_NOCACHE)#define PAGE_KERNEL_IO __pgprot(__PAGE_KERNEL_IO)#define PAGE_KERNEL_IO_NOCACHE __pgprot(__PAGE_KERNEL_IO_NOCACHE) -#define PAGE_KERNEL_IO_UC_MINUS __pgprot(__PAGE_KERNEL_IO_UC_MINUS) -#define PAGE_KERNEL_IO_WC __pgprot(__PAGE_KERNEL_IO_WC)/* xwr */ #define __P000 PAGE_NONE
arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 335
Quoted 5 lines
arch/x86/include/asm/pgtable_types.h +2
-0@@ -328,6 +331,55 @@ static inline pteval_t pte_flags(pte_t pte)#define pgprot_val (x) ((x).pgprot)#define __pgprot (x) ((pgprot_t) { (x) } ) +extern uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM]; +extern uint8_t __pte2cachemode_tbl[8 ];
You might want to use a #define for the 8 eight value. Or just have a comment saying it is to match the PAT0-7 entries.
Maybe mention SDM 11.12.3 and how the PAT, PCT, PWT play a role in determining which PATi is selected; and how the PATi memory types match up with the _PAGE_CACHE_MODE_NUM enums?
arch/x86/include/asm/pgtable_types.h static inline pteval_t pte_flags(pte_t pte) · line 340
Quoted 5 lines
arch/x86/include/asm/pgtable_types.h +5
-0+ +#define __pte2cm_idx (cb) \ + ((((cb) >> (_PAGE_BIT_PAT - 2)) & 4) | \ + (((cb) >> (_PAGE_BIT_PCD - 1)) & 2) | \ + (((cb) >> _PAGE_BIT_PWT) & 1))
Also I would say:
/* See __pte2cachemode_tbl for an example of how this is used. */
Quoted 59 lines
arch/x86/include/asm/pgtable_types.h +42
-0+ +static inline unsigned long cachemode2protval (enum page_cache_mode pcm) +{ +if (likely (pcm ==0 )) +return 0 ; +return __cachemode2pte_tbl[pcm]; +} +static inline pgprot_t cachemode2pgprot (enum page_cache_mode pcm) +{ +return __pgprot (cachemode2protval (pcm)); +} +static inline enum page_cache_mode pgprot2cachemode (pgprot_t pgprot) +{ +unsigned long masked; + + masked =pgprot_val (pgprot) & _PAGE_CACHE_MASK; +if (likely (masked ==0 )) +return 0 ; +return __pte2cachemode_tbl[__pte2cm_idx (masked)]; +} +static inline pgprot_t pgprot_4k_2_large (pgprot_t pgprot) +{ +pgprot_t new; +unsigned long val; + + val =pgprot_val (pgprot); +pgprot_val (new) = (val & ~(_PAGE_PAT | _PAGE_PAT_LARGE)) | + ((val & _PAGE_PAT) << (_PAGE_BIT_PAT_LARGE - _PAGE_BIT_PAT)); +return new; +} +static inline pgprot_t pgprot_large_2_4k (pgprot_t pgprot) +{ +pgprot_t new; +unsigned long val; + + val =pgprot_val (pgprot); +pgprot_val (new) = (val & ~(_PAGE_PAT | _PAGE_PAT_LARGE)) | + ((val & _PAGE_PAT_LARGE) >> + (_PAGE_BIT_PAT_LARGE - _PAGE_BIT_PAT)); +return new; +} +typedef struct page *pgtable_t ;arch/x86/mm/dump_pagetables.c +1
-1@@ -122,7 +122,7 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)if (!pgprot_val (prot)) {/* Not present */ -pt_dump_cont_printf (m, dmsg," " ); +pt_dump_cont_printf (m, dmsg," " ); }else {if (pr & _PAGE_USER)pt_dump_cont_printf (m, dmsg,"USR " );
arch/x86/mm/dump_pagetables.c static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg) · line 144
Quoted 4 lines
arch/x86/mm/dump_pagetables.c +0
-1@@ -141,18 +141,16 @@ static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg)else pt_dump_cont_printf (m, dmsg," " ); -/* Bit 9 has a different meaning on level 3 vs 4 */
HA!
arch/x86/mm/dump_pagetables.c static void printk_prot(struct seq_file *m, pgprot_t prot, int level, bool dmsg) · line 144
Quoted 12 lines
arch/x86/mm/dump_pagetables.c +1
-11-if (level <=3 ) { -if (pr & _PAGE_PSE) -pt_dump_cont_printf (m, dmsg,"PSE " ); -else -pt_dump_cont_printf (m, dmsg," " ); - }else { -if (pr & _PAGE_PAT) -pt_dump_cont_printf (m, dmsg,"pat " ); -else -pt_dump_cont_printf (m, dmsg," " ); - } +/* Bit 7 has a different meaning on level 3 vs 4 */
You are fixing this as well (might want to mention that in the description).
Quoted 17 lines
arch/x86/mm/dump_pagetables.c +9
-0+if (level <=3 && pr & _PAGE_PSE) +pt_dump_cont_printf (m, dmsg,"PSE " ); +else +pt_dump_cont_printf (m, dmsg," " ); +if ((level ==4 && pr & _PAGE_PAT) || + ((level ==3 || level ==2 ) && pr & _PAGE_PAT_LARGE)) +pt_dump_cont_printf (m, dmsg,"pat " ); +else +pt_dump_cont_printf (m, dmsg," " );if (pr & _PAGE_GLOBAL)pt_dump_cont_printf (m, dmsg,"GLB " );else arch/x86/mm/init.c +0
-0
arch/x86/mm/init.c line 36
Quoted 10 lines
arch/x86/mm/init.c +7
-0@@ -27,6 +27,35 @@#include "mm_internal.h" +/* +* Tables translating between page_cache_type_t and pte encoding. +* Minimal supported modes are defined statically, modified if more supported +* cache modes are available. +* Index into __cachemode2pte_tbl is the cachemode. +* Index into __pte2cachemode_tbl are the caching attribute bits of the pte +* (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
Nicely put. Could be replicated in the pte2cm_idx on how it is suppose to work.
Quoted 25 lines
arch/x86/mm/init.c +22
-0+*/ +uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = { + [_PAGE_CACHE_MODE_WB] =0 , + [_PAGE_CACHE_MODE_WC] = _PAGE_PWT, + [_PAGE_CACHE_MODE_UC_MINUS] = _PAGE_PCD, + [_PAGE_CACHE_MODE_UC] = _PAGE_PCD | _PAGE_PWT, + [_PAGE_CACHE_MODE_WT] = _PAGE_PCD, + [_PAGE_CACHE_MODE_WP] = _PAGE_PCD, +}; +EXPORT_SYMBOL_GPL (__cachemode2pte_tbl); +uint8_t __pte2cachemode_tbl[8 ] = { + [__pte2cm_idx (0 )] = _PAGE_CACHE_MODE_WB, + [__pte2cm_idx (_PAGE_PWT)] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx (_PAGE_PCD)] = _PAGE_CACHE_MODE_UC_MINUS, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PCD)] = _PAGE_CACHE_MODE_UC, + [__pte2cm_idx (_PAGE_PAT)] = _PAGE_CACHE_MODE_WB, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx (_PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS, + [__pte2cm_idx (_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC, +}; +EXPORT_SYMBOL_GPL (__pte2cachemode_tbl); +static unsigned long __initdata pgt_buf_start;static unsigned long __initdata pgt_buf_end;static unsigned long __initdata pgt_buf_top;
Juergen Gross
maintainerPATCH v3 2/3 x86: Enable PAT to use cache mode translation tables
In reply to Juergen GrossUpdate the translation tables from cache mode to pgprot values according to the PAT settings. This enables changing the cache attributes of a PAT index in just one place without having to change at the users side.
With this change it is possible to use the same kernel with different PAT configurations, e.g. supporting Xen.
--- arch/x86/include/asm/pat.h | 1 + arch/x86/include/asm/pgtable_types.h | 4 +++ arch/x86/mm/init.c | 8 ++++++ arch/x86/mm/mm_internal.h | 2 ++ arch/x86/mm/pat.c | 55 ++++++++++++++++++++++++++++++++++-- 5 files changed, 68 insertions(+), 2 deletions(-)
arch/x86/include/asm/pat.h +1 -0
@@ -11,6 +11,7 @@ static const int pat_enabled;#endif extern void pat_init (void ); +void pat_init_cache_modes (void );extern int reserve_memtype (u64 start,u64 end,enum page_cache_mode req_pcm,enum page_cache_mode *ret_pcm);
arch/x86/include/asm/pgtable_types.h +4 -0
@@ -338,6 +338,10 @@ extern uint8_t __pte2cachemode_tbl[8]; ((((cb) >> (_PAGE_BIT_PAT -2 )) &4 ) | \ (((cb) >> (_PAGE_BIT_PCD -1 )) &2 ) | \ (((cb) >> _PAGE_BIT_PWT) & 1)) +#define __cm_idx2pte (i) \ + ((((i) & 4) << (_PAGE_BIT_PAT - 2)) | \ + (((i) & 2) << (_PAGE_BIT_PCD - 1)) | \ + (((i) & 1) << _PAGE_BIT_PWT))static inline unsigned long cachemode2protval (enum page_cache_mode pcm) {
arch/x86/mm/init.c +8 -0
@@ -716,3 +716,11 @@ void __init zone_sizes_init(void)free_area_init_nodes (max_zone_pfns); } +void update_cache_mode_entry (unsigned entry,enum page_cache_mode cache) +{ +/* entry 0 MUST be WB (hardwired to speed up translations) */ +BUG_ON (!entry && cache != _PAGE_CACHE_MODE_WB); + + __cachemode2pte_tbl[cache] =__cm_idx2pte (entry); + __pte2cachemode_tbl[entry] = cache; +}
arch/x86/mm/mm_internal.h +2 -0
@@ -16,4 +16,6 @@ void zone_sizes_init(void);extern int after_bootmem; +void update_cache_mode_entry (unsigned entry,enum page_cache_mode cache); +#endif /* __X86_MM_INTERNAL_H */
arch/x86/mm/pat.c +53 -2
73 lines not on this page · raw message
Signature · 1 line
1.8.4.5
Re: [PATCH V3 2/3] x86: Enable PAT to use cache mode translation tables
Quoted 10 lines · Juergen Gross
Update the translation tables from cache mode to pgprot values according to the PAT settings. This enables changing the cache attributes of a PAT index in just one place without having to change at the users side.
With this change it is possible to use the same kernel with different PAT configurations, e.g. supporting Xen.
Signed-off-by Juergen Gross <[email protected]>Reviewed-by Toshi Kani <[email protected]>
Quoted 145 lines
--- arch/x86/include/asm/pat.h | 1 + arch/x86/include/asm/pgtable_types.h | 4 +++ arch/x86/mm/init.c | 8 ++++++ arch/x86/mm/mm_internal.h | 2 ++ arch/x86/mm/pat.c | 55 ++++++++++++++++++++++++++++++++++-- 5 files changed, 68 insertions(+), 2 deletions(-)arch/x86/include/asm/pat.h +1
-0@@ -11,6 +11,7 @@ static const int pat_enabled;#endif extern void pat_init (void ); +void pat_init_cache_modes (void );extern int reserve_memtype (u64 start,u64 end,enum page_cache_mode req_pcm,enum page_cache_mode *ret_pcm);arch/x86/include/asm/pgtable_types.h +4
-0@@ -338,6 +338,10 @@ extern uint8_t __pte2cachemode_tbl[8]; ((((cb) >> (_PAGE_BIT_PAT -2 )) &4 ) | \ (((cb) >> (_PAGE_BIT_PCD -1 )) &2 ) | \ (((cb) >> _PAGE_BIT_PWT) & 1)) +#define __cm_idx2pte (i) \ + ((((i) & 4) << (_PAGE_BIT_PAT - 2)) | \ + (((i) & 2) << (_PAGE_BIT_PCD - 1)) | \ + (((i) & 1) << _PAGE_BIT_PWT))static inline unsigned long cachemode2protval (enum page_cache_mode pcm) {arch/x86/mm/init.c +8
-0@@ -716,3 +716,11 @@ void __init zone_sizes_init(void)free_area_init_nodes (max_zone_pfns); } +void update_cache_mode_entry (unsigned entry,enum page_cache_mode cache) +{ +/* entry 0 MUST be WB (hardwired to speed up translations) */ +BUG_ON (!entry && cache != _PAGE_CACHE_MODE_WB); + + __cachemode2pte_tbl[cache] =__cm_idx2pte (entry); + __pte2cachemode_tbl[entry] = cache; +}arch/x86/mm/mm_internal.h +2
-0@@ -16,4 +16,6 @@ void zone_sizes_init(void);extern int after_bootmem; +void update_cache_mode_entry (unsigned entry,enum page_cache_mode cache); +#endif /* __X86_MM_INTERNAL_H */ arch/x86/mm/pat.c +53
-2@@ -31,6 +31,7 @@#include <asm/io.h> #include "pat_internal.h" +#include "mm_internal.h" #ifdef CONFIG_X86_PAT int __read_mostly pat_enabled =1 ;@@ -75,6 +76,57 @@ enum { PAT_UC_MINUS = 7, /* UC, but can be overriden by MTRR */ }; +/* +* Update the cache mode to pgprot translation tables according to PAT +* configuration. +* Using lower indices is preferred, so we start with highest index. +*/ +void pat_init_cache_modes (void ) +{ +int i; +enum page_cache_mode cache; +char pat_msg[33 ]; +char *cache_mode; +u64 pat; + +rdmsrl (MSR_IA32_CR_PAT , pat); + pat_msg[32 ] =0 ; +for (i =7 ; i >=0 ; i--) { +switch ((pat >> (i *8 )) &7 ) { +case PAT_UC : + cache = _PAGE_CACHE_MODE_UC; + cache_mode ="UC " ; +break ; +case PAT_WC : + cache = _PAGE_CACHE_MODE_WC; + cache_mode ="WC " ; +break ; +case PAT_WT : + cache = _PAGE_CACHE_MODE_WT; + cache_mode ="WT " ; +break ; +case PAT_WP : + cache = _PAGE_CACHE_MODE_WP; + cache_mode ="WP " ; +break ; +case PAT_WB : + cache = _PAGE_CACHE_MODE_WB; + cache_mode ="WB " ; +break ; +case PAT_UC_MINUS : + cache = _PAGE_CACHE_MODE_UC_MINUS; + cache_mode ="UC- " ; +break ; +default : + cache = _PAGE_CACHE_MODE_WB; + cache_mode ="WB " ; + } +update_cache_mode_entry (i, cache); +memcpy (pat_msg +4 * i, cache_mode,4 ); + } +pr_info ("PAT configuration [0-7]: %s\n" , pat_msg); +} +#define PAT (x, y) ((u64)PAT_ ## y << ((x)*8))void pat_init (void )@@ -124,8 +176,7 @@ void pat_init(void)wrmsrl (MSR_IA32_CR_PAT , pat);if (boot_cpu) -printk (KERN_INFO "x86 PAT enabled: cpu %d, old 0x%Lx, new 0x%Lx\n" , -smp_processor_id (), boot_pat_state, pat); +pat_init_cache_modes (); }#undef PATSignature · 1 line
1.8.4.5
Juergen Gross
maintainerPATCH v3 3/3 Support Xen pv-domains using PAT
In reply to Juergen GrossWith the dynamical mapping between cache modes and pgprot values it is now possible to use all cache modes via the Xen hypervisor PAT settings in a pv domain.
All to be done is to read the PAT configuration MSR and set up the translation tables accordingly.
--- arch/x86/xen/enlighten.c | 25 +++++++------------------ arch/x86/xen/mmu.c | 48 ++---------------------------------------------- arch/x86/xen/xen-ops.h | 1 - 3 files changed, 9 insertions(+), 65 deletions(-)
arch/x86/xen/enlighten.c +7 -18
@@ -1100,12 +1100,6 @@ static int xen_write_msr_safe(unsigned int msr, unsigned low, unsigned high)/* Fast syscall setup is all done in hypercalls, so these are all ignored. Stub them out here to stop Xen console noise. */ -break ; - -case MSR_IA32_CR_PAT : -if (smp_processor_id () ==0 ) -xen_set_pat (((u64 )high <<32 ) | low); -break ;default : ret =native_write_msr_safe (msr, low, high);
@@ -1554,10 +1548,6 @@ asmlinkage __visible void __init xen_start_kernel(void)/* Prevent unwanted bits from being set in PTEs. */ __supported_pte_mask &= ~_PAGE_GLOBAL; -#if 0 - if (!xen_initial_domain()) -#endif - __supported_pte_mask &= ~(_PAGE_PWT | _PAGE_PCD); __supported_pte_mask |= _PAGE_IOMAP;
@@ -1613,14 +1603,6 @@ asmlinkage __visible void __init xen_start_kernel(void)*/ acpi_numa =-1 ;#endif -#ifdef CONFIG_X86_PAT -/* -* For right now disable the PAT. We should remove this once -* git commit 8eaffa67b43e99ae581622c5133e20b0f48bcef1 -* (xen/pat: Disable PAT support for now) is reverted. -*/ - pat_enabled =0 ; -#endif /* Don't do the full vcpu_info placement stuff until we have a possible map and a non-dummy shared_info. */ per_cpu (xen_vcpu,0 ) = &HYPERVISOR_shared_info->vcpu_info[0 ];
@@ -1634,6 +1616,13 @@ asmlinkage __visible void __init xen_start_kernel(void)/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list (); +/* +* Modify the cache mode translation tables to match Xen's PAT +* configuration. +*/ + +pat_init_cache_modes (); +/* keep using Xen gdt for now; no urgent need to change it */ #ifdef CONFIG_X86_32
arch/x86/xen/mmu.c +2 -46
63 lines not on this page · raw message
arch/x86/xen/xen-ops.h +0 -1
@@ -33,7 +33,6 @@ extern unsigned long xen_max_p2m_pfn;void xen_mm_pin_all (void );void xen_mm_unpin_all (void ); -void xen_set_pat (u64 );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );
Signature · 1 line
1.8.4.5
Re: [PATCH V3 3/3] Support Xen pv-domains using PAT
Quoted 9 lines · Juergen Gross
With the dynamical mapping between cache modes and pgprot values it is now possible to use all cache modes via the Xen hypervisor PAT settings in a pv domain.
All to be done is to read the PAT configuration MSR and set up the translation tables accordingly.
Signed-off-by Juergen Gross <[email protected]>Reviewed-by David Vrabel <[email protected]>
Quoted 5 lines
--- arch/x86/xen/enlighten.c | 25 +++++++------------------ arch/x86/xen/mmu.c | 48 ++---------------------------------------------- arch/x86/xen/xen-ops.h | 1 - 3 files changed, 9 insertions(+), 65 deletions(-)
Oh yeah! That is nice to see :-)
Quoted 142 lines
arch/x86/xen/enlighten.c +7
-18@@ -1100,12 +1100,6 @@ static int xen_write_msr_safe(unsigned int msr, unsigned low, unsigned high)/* Fast syscall setup is all done in hypercalls, so these are all ignored. Stub them out here to stop Xen console noise. */ -break ; - -case MSR_IA32_CR_PAT : -if (smp_processor_id () ==0 ) -xen_set_pat (((u64 )high <<32 ) | low); -break ;default : ret =native_write_msr_safe (msr, low, high);@@ -1554,10 +1548,6 @@ asmlinkage __visible void __init xen_start_kernel(void)/* Prevent unwanted bits from being set in PTEs. */ __supported_pte_mask &= ~_PAGE_GLOBAL; -#if 0 - if (!xen_initial_domain()) -#endif - __supported_pte_mask &= ~(_PAGE_PWT | _PAGE_PCD); __supported_pte_mask |= _PAGE_IOMAP;@@ -1613,14 +1603,6 @@ asmlinkage __visible void __init xen_start_kernel(void)*/ acpi_numa =-1 ;#endif -#ifdef CONFIG_X86_PAT -/* -* For right now disable the PAT. We should remove this once -* git commit 8eaffa67b43e99ae581622c5133e20b0f48bcef1 -* (xen/pat: Disable PAT support for now) is reverted. -*/ - pat_enabled =0 ; -#endif /* Don't do the full vcpu_info placement stuff until we have a possible map and a non-dummy shared_info. */ per_cpu (xen_vcpu,0 ) = &HYPERVISOR_shared_info->vcpu_info[0 ];@@ -1634,6 +1616,13 @@ asmlinkage __visible void __init xen_start_kernel(void)/* Allocate and initialize top and mid mfn levels for p2m structure */ xen_build_mfn_list_list (); +/* +* Modify the cache mode translation tables to match Xen's PAT +* configuration. +*/ + +pat_init_cache_modes (); +/* keep using Xen gdt for now; no urgent need to change it */ #ifdef CONFIG_X86_32 arch/x86/xen/mmu.c +2
-46@@ -434,13 +434,7 @@ static pteval_t iomap_pte(pteval_t val) __visible pteval_t xen_pte_val(pte_t pte) {pteval_t pteval = pte.pte; -#if 0 -/* If this is a WC pte, convert back from Xen WC to Linux WC */ -if ((pteval & (_PAGE_PAT | _PAGE_PCD | _PAGE_PWT)) == _PAGE_PAT) { -WARN_ON (!pat_enabled); - pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT; - } -#endif +if (xen_initial_domain () && (pteval & _PAGE_IOMAP))return pteval;@@ -454,48 +448,10 @@ __visible pgdval_t xen_pgd_val(pgd_t pgd) }PV_CALLEE_SAVE_REGS_THUNK (xen_pgd_val); -/* -* Xen's PAT setup is part of its ABI, though I assume entries 6 & 7 -* are reserved for now, to correspond to the Intel-reserved PAT -* types. -* -* We expect Linux's PAT set as follows: -* -* Idx PTE flags Linux Xen Default -* 0 WB WB WB -* 1 PWT WC WT WT -* 2 PCD UC- UC- UC- -* 3 PCD PWT UC UC UC -* 4 PAT WB WC WB -* 5 PAT PWT WC WP WT -* 6 PAT PCD UC- rsv UC- -* 7 PAT PCD PWT UC rsv UC -*/ - -void xen_set_pat (u64 pat) -{ -/* We expect Linux to use a PAT setting of -* UC UC- WC WB (ignoring the PAT flag) */ -WARN_ON (pat !=0x0007010600070106ull ); -} - __visible pte_t xen_make_pte(pteval_t pte) {phys_addr_t addr = (pte &PTE_PFN_MASK ); -#if 0 -/* If Linux is trying to set a WC pte, then map to the Xen WC. -* If _PAGE_PAT is set, then it probably means it is really -* _PAGE_PSE, so avoid fiddling with the PAT mapping and hope -* things work out OK... -* -* (We should never see kernel mappings with _PAGE_PSE set, -* but we could see hugetlbfs mappings, I think.). -*/ -if (pat_enabled && !WARN_ON (pte & _PAGE_PAT)) { -if ((pte & (_PAGE_PCD | _PAGE_PWT)) == _PAGE_PWT) - pte = (pte & ~(_PAGE_PCD | _PAGE_PWT)) | _PAGE_PAT; - } -#endif +/* * Unprivileged domains are allowed to do IOMAPpings for * PCI passthrough, but not map ISA space. The ISA arch/x86/xen/xen-ops.h +0
-1@@ -33,7 +33,6 @@ extern unsigned long xen_max_p2m_pfn;void xen_mm_pin_all (void );void xen_mm_unpin_all (void ); -void xen_set_pat (u64 );char *__init xen_memory_setup (void );char *xen_auto_xlated_memory_setup (void );Signature · 1 line
1.8.4.5
Juergen Gross
maintainerHi,
any chance to have this in 3.18?
Juergen
Quoted 79 lines · Juergen Gross
The x86 architecture offers via the PAT (Page Attribute Table) a way to specify different caching modes in page table entries. The PAT MSR contains 8 entries each specifying one of 6 possible cache modes. A pte references one of those entries via 3 bits:
_PAGE_PAT,_PAGE_PWTand_PAGE_PCD.The Linux kernel currently supports only 4 different cache modes. The PAT MSR is set up in a way that the setting of
_PAGE_PATin a pte doesn't matter: the top 4 entries in the PAT MSR are the same as the 4 lower entries.This results in the kernel not supporting e.g. write-through mode. Especially this cache mode would speed up drivers of video cards which now have to use uncached accesses.
OTOH some old processors (Pentium) don't support PAT correctly and the Xen hypervisor has been using a different PAT MSR configuration for some time now and can't change that as this setting is part of the ABI.
This patch set abstracts the cache mode from the pte and introduces tables to translate between cache mode and pte bits (the default cache mode "write back" is hard-wired to PAT entry 0). The tables are statically initialized with values being compatible to old processors and current usage. As soon as the PAT MSR is changed (or - in case of Xen - is read at boot time) the tables are changed accordingly. Requests of mappings with special cache modes are always possible now, in case they are not supported there will be a fallback to a compatible but slower mode.
Summing it up, this patch set adds the following features:
- capability to support WT and WP cache modes on processors with full PAT support
- processors with no or uncorrect PAT support are still working as today, even if WT or WP cache mode are selected by drivers for some pages
- reduction of Xen special handling regarding cache modeChanges in V3:
- corrected two minor nits (UC_MINUS, again) detected by Toshi KaniChanges in V2:
- simplified handling of PAT MSR write under Xen as suggested by David Vrabel
- removed resetting ofpat_enabledunder Xen
- two small corrections requested by Toshi Kani (UC_MINUScache mode in vermilion driver, fix 32 bit kernel build failure)
- correct build error on non-x86 arch by moving definition ofupdate_cache_mode_entry()to x86 specific headerChanges since RFC:
- renamed functions and variables as suggested by Toshi Kani
- corrected cache mode bits for WT and WP
- modified handling of PAT MSR write under Xen as suggested by Jan BeulichJuergen Gross (3):
x86: Make page cache mode a real type
x86: Enable PAT to use cache mode translation tables Support Xen pv-domains using PATarch/x86/include/asm/cacheflush.h | 38 ++++--- arch/x86/include/asm/fb.h | 6 +- arch/x86/include/asm/io.h | 2 +- arch/x86/include/asm/pat.h | 7 +- arch/x86/include/asm/pgtable.h | 19 ++-- arch/x86/include/asm/pgtable_types.h | 96 ++++++++++++---- arch/x86/mm/dump_pagetables.c | 24 ++-- arch/x86/mm/init.c | 37 ++++++ arch/x86/mm/init_64.c | 9 +- arch/x86/mm/iomap_32.c | 15 ++- arch/x86/mm/ioremap.c | 63 ++++++----- arch/x86/mm/mm_internal.h | 2 + arch/x86/mm/pageattr.c | 84 ++++++++------ arch/x86/mm/pat.c | 181 +++++++++++++++++++----------- arch/x86/mm/pat_internal.h | 22 ++-- arch/x86/mm/pat_rbtree.c | 8 +- arch/x86/pci/i386.c | 4 +- arch/x86/xen/enlighten.c | 25 ++--- arch/x86/xen/mmu.c | 48 +------- arch/x86/xen/xen-ops.h | 1 - drivers/video/fbdev/gbefb.c | 3 +- drivers/video/fbdev/vermilion/vermilion.c | 6 +- 22 files changed, 421 insertions(+), 279 deletions(-)