Allow setting caching mode in arch_add_memory() for P2PDMA
Subsystems (11)
v4 cover Allow setting caching mode in arch_add_memory() for P2PDMA
Hi,
This is v4 of the patchset which cleans up a number of minor issues from the feedback of v3 and rebases onto v5.6-rc4. Additional feedback is welcome.
Also worth noting, is that the kernel test robot reports[1] that Patch 3 in this series improves will-it-scale.per_process_ops by 36%. Though, for the life of me, I can't understand why that would be. But it's reported the same thing twice now for different versions of the series.
Thanks,
Logan
[1] https://lists.01.org/hyperkitty/list/[email protected]/thread/5APDKNBEJGVJTJRTI2IIA3P4OC2OEYPS/
--
Changes in v4:
* Rebased onto v5.6-rc4
* Collected tags form David, Dan and Michal
* Minor changes to the new _set_memory_prot() function and added some comments as requested by Dan.
* Changed the default caching type for P2PDMA memory to UC instead of WC per Jason's concerns that WC might be more generally unsafe.
Changes in v3:
* Rebased onto v5.6-rc2
* Rename mhp_modifiers to mhp_params per David with an updated kernel doc per Dan
* Drop support for s390 per David seeing it does not support ZONE_DEVICE yet and there was a potential problem with huge pages.
* Added WARN_ON_ONCE in cases where arches recieve non PAGE_KERNEL parameters
* Collected David and Micheal's Reviewed-By and Acked-by Tags
Changes in v2:
* Rebased onto v5.5-rc5
* Renamed mhp_restrictions to mhp_modifiers and added the pgprot field to that structure instead of using an argument for arch_add_memory().
* Add patch to drop the unused flags field in mhp_restrictions
A git branch is available here:
https://github.com/sbates130272/linux-p2pmem remap_pages_cache_v4
--
Currently, the page tables created using memremap_pages() are always created with the PAGE_KERNEL cacheing mode. However, the P2PDMA code is creating pages for PCI BAR memory which should never be accessed through the cache and instead use either WC or UC. This still works in most cases, on x86, because the MTRR registers typically override the caching settings in the page tables for all of the IO memory to be UC-. However, this tends not to work so well on other arches or some rare x86 machines that have firmware which does not setup the MTRR registers in this way.
Instead of this, this series proposes a change to arch_add_memory() to take the pgprot required by the mapping which allows us to explicitly set pagetable entries for P2PDMA memory to UC.
This changes is pretty routine for most of the arches: x86_64, arm64 and powerpc simply need to thread the pgprot through to where the page tables are setup. x86_32 unfortunately sets up the page tables at boot so must use _set_memory_prot() to change their caching mode. ia64, s390 and sh don't appear to have an easy way to change the page tables so, for now at least, we just return -EINVAL on such mappings and thus they will not support P2PDMA memory until the work for this is done. This should be fine as they don't yet support ZONE_DEVICE.
--
Logan Gunthorpe (7):
mm/memory_hotplug: Drop the flags field from struct mhp_restrictions mm/memory_hotplug: Rename mhp_restrictions to mhp_params x86/mm: Thread pgprot_t through init_memory_mapping() x86/mm: Introduce __set_memory_prot()
powerpc/mm: Thread pgprot_t through create_section_mapping() mm/memory_hotplug: Add pgprot_t to mhp_params mm/memremap: Set caching mode for PCI P2PDMA memory to WC
arch/arm64/mm/mmu.c | 7 ++-- arch/ia64/mm/init.c | 7 ++-- arch/powerpc/include/asm/book3s/64/hash.h | 3 +- arch/powerpc/include/asm/book3s/64/radix.h | 3 +- arch/powerpc/include/asm/sparsemem.h | 3 +- arch/powerpc/mm/book3s64/hash_utils.c | 5 +-- arch/powerpc/mm/book3s64/pgtable.c | 7 ++-- arch/powerpc/mm/book3s64/radix_pgtable.c | 18 ++++++---- arch/powerpc/mm/mem.c | 10 +++--- arch/s390/mm/init.c | 9 +++-- arch/sh/mm/init.c | 7 ++-- arch/x86/include/asm/page_types.h | 3 -- arch/x86/include/asm/pgtable.h | 3 ++ arch/x86/include/asm/set_memory.h | 1 + arch/x86/kernel/amd_gart_64.c | 3 +- arch/x86/mm/init.c | 9 ++--- arch/x86/mm/init_32.c | 19 ++++++++-- arch/x86/mm/init_64.c | 40 ++++++++++++---------- arch/x86/mm/mm_internal.h | 3 +- arch/x86/mm/pat/set_memory.c | 13 +++++++ arch/x86/platform/uv/bios_uv.c | 3 +- include/linux/memory_hotplug.h | 21 ++++++------ mm/memory_hotplug.c | 11 +++--- mm/memremap.c | 17 +++++---- 24 files changed, 144 insertions(+), 81 deletions(-)
base-commit: 98d54f81e36ba3bf92172791eba5ca5bd813989b --
2.20.1
PATCH v4 1/7 mm/memory_hotplug: Drop the flags field from struct mhp_restrictions
This variable is not used anywhere and should therefore be removed from the structure.
--- include/linux/memory_hotplug.h | 2 -- 1 file changed, 2 deletions(-)
include/linux/memory_hotplug.h +0 -2
@@ -55,11 +55,9 @@ enum {/* * Restrictions for the memory hotplug: -* flags: MHP_ flags * altmap: alternative allocator for memmap array */ struct mhp_restrictions { -unsigned long flags;struct vmem_altmap *altmap; };
Signature · 1 line
2.20.1
PATCH v4 2/7 mm/memory_hotplug: Rename mhp_restrictions to mhp_params
In reply to Logan GunthorpeThe mhp_restrictions struct really doesn't specify anything resembling a restriction anymore so rename it to be mhp_params as it is a list of extended parameters.
--- arch/arm64/mm/mmu.c | 4 ++-- arch/ia64/mm/init.c | 4 ++-- arch/powerpc/mm/mem.c | 4 ++-- arch/s390/mm/init.c | 6 +++--- arch/sh/mm/init.c | 4 ++-- arch/x86/mm/init_32.c | 4 ++-- arch/x86/mm/init_64.c | 8 ++++---- include/linux/memory_hotplug.h | 16 ++++++++-------- mm/memory_hotplug.c | 8 ++++---- mm/memremap.c | 8 ++++---- 10 files changed, 33 insertions(+), 33 deletions(-)
arch/arm64/mm/mmu.c +2 -2
@@ -1050,7 +1050,7 @@ int p4d_free_pud_page(p4d_t *p4d, unsigned long addr)#ifdef CONFIG_MEMORY_HOTPLUG int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {int flags =0 ;
@@ -1063,7 +1063,7 @@ int arch_add_memory(int nid, u64 start, u64 size,memblock_clear_nomap (start, size);return __add_pages (nid, start >>PAGE_SHIFT , size >>PAGE_SHIFT , - restrictions); + params); }void arch_remove_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap)
arch/ia64/mm/init.c +2 -2
@@ -670,13 +670,13 @@ mem_init (void)#ifdef CONFIG_MEMORY_HOTPLUG int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;int ret; - ret =__add_pages (nid, start_pfn, nr_pages, restrictions); + ret =__add_pages (nid, start_pfn, nr_pages, params);if (ret)printk ("%s: Problem encountered in __add_pages() as ret=%d\n" , __func__, ret);
arch/powerpc/mm/mem.c +2 -2
@@ -128,7 +128,7 @@ static void flush_dcache_range_chunked(unsigned long start, unsigned long stop, }int __ref arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;
@@ -144,7 +144,7 @@ int __ref arch_add_memory(int nid, u64 start, u64 size,return -EFAULT ; } -return __add_pages (nid, start_pfn, nr_pages, restrictions); +return __add_pages (nid, start_pfn, nr_pages, params); } void __ref arch_remove_memory(int nid, u64 start, u64 size,
arch/s390/mm/init.c +3 -3
@@ -268,20 +268,20 @@ device_initcall(s390_cma_mem_init);#endif /* CONFIG_CMA */ int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn =PFN_DOWN (start);unsigned long size_pages =PFN_DOWN (size);int rc; -if (WARN_ON_ONCE (restrictions->altmap)) +if (WARN_ON_ONCE (params->altmap))return -EINVAL ; rc =vmem_add_mapping (start, size);if (rc)return rc; - rc =__add_pages (nid, start_pfn, size_pages, restrictions); + rc =__add_pages (nid, start_pfn, size_pages, params);if (rc)vmem_remove_mapping (start, size);return rc;
arch/sh/mm/init.c +2 -2
@@ -406,14 +406,14 @@ void __init mem_init(void)#ifdef CONFIG_MEMORY_HOTPLUG int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn =PFN_DOWN (start);unsigned long nr_pages = size >>PAGE_SHIFT ;int ret;/* We only have ZONE_NORMAL, so this is easy.. */ - ret =__add_pages (nid, start_pfn, nr_pages, restrictions); + ret =__add_pages (nid, start_pfn, nr_pages, params);if (unlikely (ret))printk ("%s: Failed, __add_pages() = %d\n" , __func__, ret);
arch/x86/mm/init_32.c +2 -2
@@ -853,12 +853,12 @@ void __init mem_init(void)#ifdef CONFIG_MEMORY_HOTPLUG int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; -return __add_pages (nid, start_pfn, nr_pages, restrictions); +return __add_pages (nid, start_pfn, nr_pages, params); } void arch_remove_memory(int nid, u64 start, u64 size,
arch/x86/mm/init_64.c +4 -4
@@ -844,11 +844,11 @@ static void update_end_of_memory_vars(u64 start, u64 size) }int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {int ret; - ret =__add_pages (nid, start_pfn, nr_pages, restrictions); + ret =__add_pages (nid, start_pfn, nr_pages, params);WARN_ON_ONCE (ret);/* update max_pfn, max_low_pfn and high_memory */
@@ -859,14 +859,14 @@ int add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages, }int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;init_memory_mapping (start, start + size); -return add_pages (nid, start_pfn, nr_pages, restrictions); +return add_pages (nid, start_pfn, nr_pages, params); }#define PAGE_INUSE 0xFD
include/linux/memory_hotplug.h +8 -8
@@ -54,10 +54,10 @@ enum { };/* -* Restrictions for the memory hotplug: -* altmap: alternative allocator for memmap array +* Extended parameters for memory hotplug: +* altmap: alternative allocator for memmap array (optional) */ -struct mhp_restrictions { +struct mhp_params {struct vmem_altmap *altmap; };
@@ -108,7 +108,7 @@ extern int restore_online_page_callback(online_page_callback_t callback);extern int try_online_node (int nid);extern int arch_add_memory (int nid,u64 start,u64 size, -struct mhp_restrictions *restrictions); +struct mhp_params *params);extern u64 max_mem_size;extern bool memhp_auto_online;
@@ -126,17 +126,17 @@ extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,/* reasonably generic interface to expand the physical pages */ extern int __add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, -struct mhp_restrictions *restrictions); +struct mhp_params *params);#ifndef CONFIG_ARCH_HAS_ADD_PAGES static inline int add_pages (int nid,unsigned long start_pfn, -unsigned long nr_pages,struct mhp_restrictions *restrictions) +unsigned long nr_pages,struct mhp_params *params) { -return __add_pages (nid, start_pfn, nr_pages, restrictions); +return __add_pages (nid, start_pfn, nr_pages, params); }#else /* ARCH_HAS_ADD_PAGES */ int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, -struct mhp_restrictions *restrictions); +struct mhp_params *params);#endif /* ARCH_HAS_ADD_PAGES */ #ifdef CONFIG_NUMA
mm/memory_hotplug.c +4 -4
@@ -299,11 +299,11 @@ static int check_hotplug_memory_addressable(unsigned long pfn,* add the new pages. */ int __ref __add_pages (int nid,unsigned long pfn,unsigned long nr_pages, -struct mhp_restrictions *restrictions) +struct mhp_params *params) {int err;unsigned long nr, start_sec, end_sec; -struct vmem_altmap *altmap = restrictions->altmap; +struct vmem_altmap *altmap = params->altmap; err =check_hotplug_memory_addressable (pfn, nr_pages);if (err)
@@ -993,7 +993,7 @@ static int online_memory_block(struct memory_block *mem, void *arg)*/ int __ref add_memory_resource (int nid,struct resource *res) { -struct mhp_restrictions restrictions = {}; +struct mhp_params params = {};u64 start, size;bool new_node = false;int ret;
@@ -1021,7 +1021,7 @@ int __ref add_memory_resource(int nid, struct resource *res) new_node = ret;/* call arch's memory hotadd */ - ret =arch_add_memory (nid, start, size, &restrictions); + ret =arch_add_memory (nid, start, size, ¶ms);if (ret <0 ) goto error;
mm/memremap.c +4 -4
@@ -161,7 +161,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) {struct resource *res = &pgmap->res;struct dev_pagemap *conflict_pgmap; -struct mhp_restrictions restrictions = { +struct mhp_params params = {/* * We do not want any optional features only our own memmap */
@@ -275,7 +275,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)*/ if (pgmap->type =MEMORY_DEVICE_PRIVATE ) { error =add_pages (nid,PHYS_PFN (res->start), -PHYS_PFN (resource_size (res)), &restrictions); +PHYS_PFN (resource_size (res)), ¶ms); }else { error =kasan_add_zero_shadow (__va (res->start),resource_size (res));if (error) {
@@ -284,7 +284,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) } error =arch_add_memory (nid, res->start,resource_size (res), - &restrictions); + ¶ms); }if (!error) {
@@ -292,7 +292,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) zone = &NODE_DATA (nid)->node_zones[ZONE_DEVICE ];move_pfn_range_to_zone (zone,PHYS_PFN (res->start), -PHYS_PFN (resource_size (res)), restrictions.altmap); +PHYS_PFN (resource_size (res)), params.altmap); }mem_hotplug_done ();
Signature · 1 line
2.20.1
PATCH v4 3/7 x86/mm: Thread pgprot_t through init_memory_mapping()
In reply to Logan GunthorpeIn prepartion to support a pgprot_t argument for arch_add_memory().
It's required to move the prototype of init_memory_mapping() seeing the original location came before the definition of pgprot_t.
--- arch/x86/include/asm/page_types.h | 3 --- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/kernel/amd_gart_64.c | 3 ++- arch/x86/mm/init.c | 9 +++++---- arch/x86/mm/init_32.c | 3 ++- arch/x86/mm/init_64.c | 32 +++++++++++++++++-------------- arch/x86/mm/mm_internal.h | 3 ++- arch/x86/platform/uv/bios_uv.c | 3 ++- 8 files changed, 34 insertions(+), 25 deletions(-)
arch/x86/include/asm/page_types.h +0 -3
@@ -73,9 +73,6 @@ static inline phys_addr_t get_max_mapped(void)bool pfn_range_is_mapped (unsigned long start_pfn,unsigned long end_pfn); -extern unsigned long init_memory_mapping (unsigned long start, -unsigned long end); -extern void initmem_init (void );#endif /* !__ASSEMBLY__ */
arch/x86/include/asm/pgtable.h +3 -0
@@ -1046,6 +1046,9 @@ static inline void __meminit init_trampoline_default(void)void __init poking_init (void ); +unsigned long init_memory_mapping (unsigned long start, +unsigned long end,pgprot_t prot); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else
arch/x86/kernel/amd_gart_64.c +2 -1
@@ -744,7 +744,8 @@ int __init gart_iommu_init(void) start_pfn =PFN_DOWN (aper_base);if (!pfn_range_is_mapped (start_pfn, end_pfn)) -init_memory_mapping (start_pfn<<PAGE_SHIFT , end_pfn<<PAGE_SHIFT ); +init_memory_mapping (start_pfn<<PAGE_SHIFT , end_pfn<<PAGE_SHIFT , +PAGE_KERNEL );pr_info ("PCI-DMA: using GART IOMMU.\n" ); iommu_size =check_iommu_size (info.aper_base, aper_size);
arch/x86/mm/init.c +5 -4
@@ -467,7 +467,7 @@ bool pfn_range_is_mapped(unsigned long start_pfn, unsigned long end_pfn)* the physical memory. To access them they are temporarily mapped. */ unsigned long __ref init_memory_mapping (unsigned long start, -unsigned long end) +unsigned long end,pgprot_t prot) {struct map_range mr[NR_RANGE_MR ];unsigned long ret =0 ;
@@ -481,7 +481,8 @@ unsigned long __ref init_memory_mapping(unsigned long start,for (i =0 ; i < nr_range; i++) ret =kernel_physical_mapping_init (mr[i].start, mr[i].end, - mr[i].page_size_mask); + mr[i].page_size_mask, + prot);add_pfn_range_mapped (start >>PAGE_SHIFT , ret >>PAGE_SHIFT );
@@ -521,7 +522,7 @@ static unsigned long __init init_range_memory_mapping(*/ can_use_brk_pgt =max (start, (u64 )pgt_buf_end<<PAGE_SHIFT ) >min (end, (u64 )pgt_buf_top<<PAGE_SHIFT ); -init_memory_mapping (start, end); +init_memory_mapping (start, end,PAGE_KERNEL ); mapped_ram_size += end - start; can_use_brk_pgt = true; }
@@ -661,7 +662,7 @@ void __init init_mem_mapping(void)#endif /* the ISA range is always mapped regardless of memory holes */ -init_memory_mapping (0 ,ISA_END_ADDRESS ); +init_memory_mapping (0 ,ISA_END_ADDRESS ,PAGE_KERNEL );/* Init the trampoline, possibly with KASLR memory offset */ init_trampoline ();
arch/x86/mm/init_32.c +2 -1
@@ -253,7 +253,8 @@ static inline int is_kernel_text(unsigned long addr)unsigned long __init kernel_physical_mapping_init (unsigned long start,unsigned long end, -unsigned long page_size_mask) +unsigned long page_size_mask, +pgprot_t prot) {int use_pse = page_size_mask = (1 <<PG_LEVEL_2M );unsigned long last_map_addr = end;
arch/x86/mm/init_64.c +18 -14
@@ -585,7 +585,7 @@ phys_pmd_init(pmd_t *pmd_page, unsigned long paddr, unsigned long paddr_end,*/ static unsigned long __meminit phys_pud_init (pud_t *pud_page,unsigned long paddr,unsigned long paddr_end, -unsigned long page_size_mask,bool init) +unsigned long page_size_mask,pgprot_t _prot,bool init) {unsigned long pages =0 , paddr_next;unsigned long paddr_last = paddr_end;
@@ -595,7 +595,7 @@ phys_pud_init(pud_t *pud_page, unsigned long paddr, unsigned long paddr_end,for (; i <PTRS_PER_PUD ; i++, paddr = paddr_next) {pud_t *pud;pmd_t *pmd; -pgprot_t prot =PAGE_KERNEL ; +pgprot_t prot = _prot; vaddr = (unsigned long )__va (paddr); pud = pud_page +pud_index (vaddr);
@@ -644,9 +644,12 @@ phys_pud_init(pud_t *pud_page, unsigned long paddr, unsigned long paddr_end,if (page_size_mask & (1 <<PG_LEVEL_1G )) { pages++;spin_lock (&init_mm.page_table_lock); + + prot =__pgprot (pgprot_val (prot) | __PAGE_KERNEL_LARGE); +set_pte_init ((pte_t *)pud,pfn_pte ((paddr &PUD_MASK ) >>PAGE_SHIFT , -PAGE_KERNEL_LARGE ), + prot), init);spin_unlock (&init_mm.page_table_lock); paddr_last = paddr_next;
@@ -669,7 +672,7 @@ phys_pud_init(pud_t *pud_page, unsigned long paddr, unsigned long paddr_end,static unsigned long __meminit phys_p4d_init (p4d_t *p4d_page,unsigned long paddr,unsigned long paddr_end, -unsigned long page_size_mask,bool init) +unsigned long page_size_mask,pgprot_t prot,bool init) {unsigned long vaddr, vaddr_end, vaddr_next, paddr_next, paddr_last;
@@ -679,7 +682,7 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end,if (!pgtable_l5_enabled ())return phys_pud_init ((pud_t *) p4d_page, paddr, paddr_end, - page_size_mask, init); + page_size_mask, prot, init);for (; vaddr < vaddr_end; vaddr = vaddr_next) {p4d_t *p4d = p4d_page +p4d_index (vaddr);
@@ -702,13 +705,13 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end,if (!p4d_none (*p4d)) { pud =pud_offset (p4d,0 ); paddr_last =phys_pud_init (pud, paddr,__pa (vaddr_end), - page_size_mask, init); + page_size_mask, prot, init);continue ; } pud =alloc_low_page (); paddr_last =phys_pud_init (pud, paddr,__pa (vaddr_end), - page_size_mask, init); + page_size_mask, prot, init);spin_lock (&init_mm.page_table_lock);p4d_populate_init (&init_mm, p4d, pud, init);
@@ -722,7 +725,7 @@ static unsigned long __meminit __kernel_physical_mapping_init(unsigned long paddr_start, unsigned long paddr_end, unsigned long page_size_mask, - bool init) + pgprot_t prot, bool init) { bool pgd_changed = false;unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last;
@@ -743,13 +746,13 @@ __kernel_physical_mapping_init(unsigned long paddr_start, paddr_last =phys_p4d_init (p4d,__pa (vaddr),__pa (vaddr_end), page_size_mask, - init); + prot, init);continue ; } p4d =alloc_low_page (); paddr_last =phys_p4d_init (p4d,__pa (vaddr),__pa (vaddr_end), - page_size_mask, init); + page_size_mask, prot, init);spin_lock (&init_mm.page_table_lock);if (pgtable_l5_enabled ())
@@ -778,10 +781,10 @@ __kernel_physical_mapping_init(unsigned long paddr_start,unsigned long __meminit kernel_physical_mapping_init (unsigned long paddr_start,unsigned long paddr_end, -unsigned long page_size_mask) +unsigned long page_size_mask,pgprot_t prot) {return __kernel_physical_mapping_init (paddr_start, paddr_end, - page_size_mask, true); + page_size_mask, prot, true); }/*
@@ -796,7 +799,8 @@ kernel_physical_mapping_change(unsigned long paddr_start, unsigned long page_size_mask) {return __kernel_physical_mapping_init (paddr_start, paddr_end, - page_size_mask, false); + page_size_mask,PAGE_KERNEL , + false); }#ifndef CONFIG_NUMA
@@ -864,7 +868,7 @@ int arch_add_memory(int nid, u64 start, u64 size,unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; -init_memory_mapping (start, start + size); +init_memory_mapping (start, start + size,PAGE_KERNEL );return add_pages (nid, start_pfn, nr_pages, params); }
arch/x86/mm/mm_internal.h +2 -1
@@ -12,7 +12,8 @@ void early_ioremap_page_table_range_init(void);unsigned long kernel_physical_mapping_init (unsigned long start,unsigned long end, -unsigned long page_size_mask); +unsigned long page_size_mask, +pgprot_t prot);unsigned long kernel_physical_mapping_change (unsigned long start,unsigned long end,unsigned long page_size_mask);
arch/x86/platform/uv/bios_uv.c +2 -1
@@ -352,7 +352,8 @@ void __iomem *__init efi_ioremap(unsigned long phys_addr, unsigned long size,if (type =EFI_MEMORY_MAPPED_IO )return ioremap (phys_addr, size); - last_map_pfn =init_memory_mapping (phys_addr, phys_addr + size); + last_map_pfn =init_memory_mapping (phys_addr, phys_addr + size, +PAGE_KERNEL );if ((last_map_pfn <<PAGE_SHIFT ) < phys_addr + size) {unsigned long top = last_map_pfn <<PAGE_SHIFT ;efi_ioremap (top, size - (top - phys_addr), type, attribute);
Signature · 1 line
2.20.1
PATCH v4 4/7 x86/mm: Introduce __set_memory_prot()
In reply to Logan GunthorpeFor use in the 32bit arch_add_memory() to set the pgprot type of the memory to add.
--- arch/x86/include/asm/set_memory.h | 1 + arch/x86/mm/pat/set_memory.c | 13 +++++++++++++ 2 files changed, 14 insertions(+)
arch/x86/include/asm/set_memory.h +1 -0
@@ -34,6 +34,7 @@* The caller is required to take care of these. */ +int __set_memory_prot (unsigned long addr,int numpages,pgprot_t prot);int _set_memory_uc (unsigned long addr,int numpages);int _set_memory_wc (unsigned long addr,int numpages);int _set_memory_wt (unsigned long addr,int numpages);
arch/x86/mm/pat/set_memory.c +13 -0
@@ -1792,6 +1792,19 @@ static inline int cpa_clear_pages_array(struct page **pages, int numpages, CPA_PAGES_ARRAY, pages); } +/* +* _set_memory_prot is an internal helper for callers that have been passed +* a pgprot_t value from upper layers and a reservation has already been taken. +* If you want to set the pgprot to a specific page protocol, use the +* set_memory_xx() functions. +*/ +int __set_memory_prot (unsigned long addr,int numpages,pgprot_t prot) +{ +return change_page_attr_set_clr (&addr, numpages, prot, +__pgprot (~pgprot_val (prot)),0 ,0 , +NULL ); +} +int _set_memory_uc (unsigned long addr,int numpages) {/*
Signature · 1 line
2.20.1
PATCH v4 5/7 powerpc/mm: Thread pgprot_t through create_section_mapping()
In reply to Logan GunthorpeIn prepartion to support a pgprot_t argument for arch_add_memory().
--- arch/powerpc/include/asm/book3s/64/hash.h | 3 ++- arch/powerpc/include/asm/book3s/64/radix.h | 3 ++- arch/powerpc/include/asm/sparsemem.h | 3 ++- arch/powerpc/mm/book3s64/hash_utils.c | 5 +++-- arch/powerpc/mm/book3s64/pgtable.c | 7 ++++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 18 +++++++++++------- arch/powerpc/mm/mem.c | 5 +++-- 7 files changed, 27 insertions(+), 17 deletions(-)
arch/powerpc/include/asm/book3s/64/hash.h +2 -1
@@ -251,7 +251,8 @@ extern int __meminit hash__vmemmap_create_mapping(unsigned long start,extern void hash__vmemmap_remove_mapping (unsigned long start,unsigned long page_size); -int hash__create_section_mapping (unsigned long start,unsigned long end,int nid); +int hash__create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot);int hash__remove_section_mapping (unsigned long start,unsigned long end);#endif /* !__ASSEMBLY__ */
arch/powerpc/include/asm/book3s/64/radix.h +2 -1
@@ -289,7 +289,8 @@ static inline unsigned long radix__get_tree_size(void) }#ifdef CONFIG_MEMORY_HOTPLUG -int radix__create_section_mapping (unsigned long start,unsigned long end,int nid); +int radix__create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot);int radix__remove_section_mapping (unsigned long start,unsigned long end);#endif /* CONFIG_MEMORY_HOTPLUG */ #endif /* __ASSEMBLY__ */
arch/powerpc/include/asm/sparsemem.h +2 -1
@@ -13,7 +13,8 @@#endif /* CONFIG_SPARSEMEM */ #ifdef CONFIG_MEMORY_HOTPLUG -extern int create_section_mapping (unsigned long start,unsigned long end,int nid); +extern int create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot);extern int remove_section_mapping (unsigned long start,unsigned long end);#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/mm/book3s64/hash_utils.c +3 -2
@@ -809,7 +809,8 @@ int resize_hpt_for_hotplug(unsigned long new_mem_size)return 0 ; } -int hash__create_section_mapping (unsigned long start,unsigned long end,int nid) +int hash__create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot) {int rc;
@@ -819,7 +820,7 @@ int hash__create_section_mapping(unsigned long start, unsigned long end, int nid } rc =htab_bolt_mapping (start, end,__pa (start), -pgprot_val (PAGE_KERNEL ), mmu_linear_psize, +pgprot_val (prot), mmu_linear_psize, mmu_kernel_ssize);if (rc <0 ) {
arch/powerpc/mm/book3s64/pgtable.c +4 -3
@@ -171,12 +171,13 @@ void mmu_cleanup_all(void) }#ifdef CONFIG_MEMORY_HOTPLUG -int __meminit create_section_mapping (unsigned long start,unsigned long end,int nid) +int __meminit create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot) {if (radix_enabled ()) -return radix__create_section_mapping (start, end, nid); +return radix__create_section_mapping (start, end, nid, prot); -return hash__create_section_mapping (start, end, nid); +return hash__create_section_mapping (start, end, nid, prot); }int __meminit remove_section_mapping (unsigned long start,unsigned long end)
arch/powerpc/mm/book3s64/radix_pgtable.c +11 -7
@@ -253,7 +253,7 @@ static unsigned long next_boundary(unsigned long addr, unsigned long end)static int __meminit create_physical_mapping (unsigned long start,unsigned long end, -int nid) +int nid,pgprot_t _prot) {unsigned long vaddr, addr, mapping_size =0 ;bool prev_exec, exec = false;
@@ -289,7 +289,7 @@ static int __meminit create_physical_mapping(unsigned long start, prot =PAGE_KERNEL_X ; exec = true; }else { - prot =PAGE_KERNEL ; + prot = _prot; exec = false; }
@@ -333,7 +333,7 @@ static void __init radix_init_pgtable(void)WARN_ON (create_physical_mapping (reg->base, reg->base + reg->size, --1 )); +-1 ,PAGE_KERNEL )); }/* Find out how many PID bits are supported */
@@ -712,8 +712,10 @@ static int __meminit stop_machine_change_mapping(void *data)spin_unlock (&init_mm.page_table_lock);pte_clear (&init_mm, params->aligned_start, params->pte); -create_physical_mapping (__pa (params->aligned_start),__pa (params->start),-1 ); -create_physical_mapping (__pa (params->end),__pa (params->aligned_end),-1 ); +create_physical_mapping (__pa (params->aligned_start), +__pa (params->start),-1 ,PAGE_KERNEL ); +create_physical_mapping (__pa (params->end),__pa (params->aligned_end), +-1 ,PAGE_KERNEL );spin_lock (&init_mm.page_table_lock);return 0 ; }
@@ -870,14 +872,16 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)radix__flush_tlb_kernel_range (start, end); } -int __meminit radix__create_section_mapping (unsigned long start,unsigned long end,int nid) +int __meminit radix__create_section_mapping (unsigned long start, +unsigned long end,int nid, +pgprot_t prot) {if (end >=RADIX_VMALLOC_START ) {pr_warn ("Outside the supported range\n" );return -1 ; } -return create_physical_mapping (__pa (start),__pa (end), nid); +return create_physical_mapping (__pa (start),__pa (end), nid, prot); }int __meminit radix__remove_section_mapping (unsigned long start,unsigned long end)
arch/powerpc/mm/mem.c +3 -2
@@ -96,7 +96,8 @@ int memory_add_physaddr_to_nid(u64 start) }#endif -int __weak create_section_mapping (unsigned long start,unsigned long end,int nid) +int __weak create_section_mapping (unsigned long start,unsigned long end, +int nid,pgprot_t prot) {return -ENODEV ; }
@@ -137,7 +138,7 @@ int __ref arch_add_memory(int nid, u64 start, u64 size,resize_hpt_for_hotplug (memblock_phys_mem_size ()); start = (unsigned long )__va (start); - rc =create_section_mapping (start, start + size, nid); + rc =create_section_mapping (start, start + size, nid,PAGE_KERNEL );if (rc) {pr_warn ("Unable to create mapping for hot added memory 0x%llx..0x%llx: %d\n" , start, start + size, rc);
Signature · 1 line
2.20.1
PATCH v4 6/7 mm/memory_hotplug: Add pgprot_t to mhp_params
In reply to Logan Gunthorpedevm_memremap_pages() is currently used by the PCI P2PDMA code to create struct page mappings for IO memory. At present, these mappings are created with PAGE_KERNEL which implies setting the PAT bits to be WB. However, on x86, an mtrr register will typically override this and force the cache type to be UC-. In the case firmware doesn't set this register it is effectively WB and will typically result in a machine check exception when it's accessed.
Other arches are not currently likely to function correctly seeing they don't have any MTRR registers to fall back on.
To solve this, provide a way to specify the pgprot value explicitly to arch_add_memory().
Of the arches that support MEMORY_HOTPLUG: x86_64, and arm64 need a simple change to pass the pgprot_t down to their respective functions which set up the page tables. For x86_32, set the page tables explicitly using _set_memory_prot() (seeing they are already mapped). For ia64, s390 and sh, reject anything but PAGE_KERNEL settings -- this should be fine, for now, seeing these architectures don't support ZONE_DEVICE.
A check in __add_pages() is also added to ensure the pgprot parameter was set for all arches.
--- arch/arm64/mm/mmu.c | 3 ++- arch/ia64/mm/init.c | 3 +++ arch/powerpc/mm/mem.c | 3 ++- arch/s390/mm/init.c | 3 +++ arch/sh/mm/init.c | 3 +++ arch/x86/mm/init_32.c | 12 ++++++++++++ arch/x86/mm/init_64.c | 2 +- include/linux/memory_hotplug.h | 3 +++ mm/memory_hotplug.c | 5 ++++- mm/memremap.c | 6 +++--- 10 files changed, 36 insertions(+), 7 deletions(-)
arch/arm64/mm/mmu.c +2 -1
@@ -1058,7 +1058,8 @@ int arch_add_memory(int nid, u64 start, u64 size, flags =NO_BLOCK_MAPPINGS |NO_CONT_MAPPINGS ;__create_pgd_mapping (swapper_pg_dir, start,__phys_to_virt (start), - size,PAGE_KERNEL , __pgd_pgtable_alloc, flags); + size, params->pgprot, __pgd_pgtable_alloc, + flags);memblock_clear_nomap (start, size);
arch/ia64/mm/init.c +3 -0
@@ -676,6 +676,9 @@ int arch_add_memory(int nid, u64 start, u64 size,unsigned long nr_pages = size >>PAGE_SHIFT ;int ret; +if (WARN_ON_ONCE (params->pgprot.pgprot !=PAGE_KERNEL .pgprot)) +return -EINVAL ; + ret =__add_pages (nid, start_pfn, nr_pages, params);if (ret) printk("%s: Problem encountered in __add_pages() as ret=%d\n",
arch/powerpc/mm/mem.c +2 -1
@@ -138,7 +138,8 @@ int __ref arch_add_memory(int nid, u64 start, u64 size,resize_hpt_for_hotplug (memblock_phys_mem_size ()); start = (unsigned long )__va (start); - rc =create_section_mapping (start, start + size, nid,PAGE_KERNEL ); + rc =create_section_mapping (start, start + size, nid, + params->pgprot);if (rc) {pr_warn ("Unable to create mapping for hot added memory 0x%llx..0x%llx: %d\n" , start, start + size, rc);
arch/s390/mm/init.c +3 -0
@@ -277,6 +277,9 @@ int arch_add_memory(int nid, u64 start, u64 size,if (WARN_ON_ONCE (params->altmap))return -EINVAL ; +if (WARN_ON_ONCE (params->pgprot.pgprot !=PAGE_KERNEL .pgprot)) +return -EINVAL ; + rc =vmem_add_mapping (start, size);if (rc)return rc;
arch/sh/mm/init.c +3 -0
@@ -412,6 +412,9 @@ int arch_add_memory(int nid, u64 start, u64 size,unsigned long nr_pages = size >>PAGE_SHIFT ;int ret; + if (WARN_ON_ONCE(params->pgprot.pgprot != PAGE_KERNEL.pgprot) +return -EINVAL ; +/* We only have ZONE_NORMAL, so this is easy.. */ ret =__add_pages (nid, start_pfn, nr_pages, params);if (unlikely (ret))
arch/x86/mm/init_32.c +12 -0
@@ -858,6 +858,18 @@ int arch_add_memory(int nid, u64 start, u64 size, {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; +int ret; + +/* +* The page tables were already mapped at boot so if the caller +* requests a different mapping type then we must change all the +* pages with __set_memory_prot(). +*/ +if (params->pgprot.pgprot !=PAGE_KERNEL .pgprot) { + ret =__set_memory_prot (start, nr_pages, params->pgprot); +if (ret) +return ret; + }return __add_pages (nid, start_pfn, nr_pages, params); }
arch/x86/mm/init_64.c +1 -1
@@ -868,7 +868,7 @@ int arch_add_memory(int nid, u64 start, u64 size,unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; -init_memory_mapping (start, start + size,PAGE_KERNEL ); +init_memory_mapping (start, start + size, params->pgprot);return add_pages (nid, start_pfn, nr_pages, params); }
include/linux/memory_hotplug.h +3 -0
@@ -56,9 +56,12 @@ enum {/* * Extended parameters for memory hotplug: * altmap: alternative allocator for memmap array (optional) +* pgprot: page protection flags to apply to newly created page tables +* (required) */ struct mhp_params {struct vmem_altmap *altmap; +pgprot_t pgprot; };/*
mm/memory_hotplug.c +4 -1
@@ -305,6 +305,9 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages,unsigned long nr, start_sec, end_sec;struct vmem_altmap *altmap = params->altmap; +if (WARN_ON_ONCE (!params->pgprot.pgprot)) +return -EINVAL ; + err =check_hotplug_memory_addressable (pfn, nr_pages);if (err)return err;
@@ -993,7 +996,7 @@ static int online_memory_block(struct memory_block *mem, void *arg)*/ int __ref add_memory_resource (int nid,struct resource *res) { -struct mhp_params params = {}; +struct mhp_params params = { .pgprot =PAGE_KERNEL };u64 start, size;bool new_node = false;int ret;
mm/memremap.c +3 -3
@@ -166,8 +166,8 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)* We do not want any optional features only our own memmap */ .altmap = pgmap_altmap(pgmap), + .pgprot = PAGE_KERNEL, }; -pgprot_t pgprot =PAGE_KERNEL ;int error, is_ram;bool need_devmap_managed = true;
@@ -255,8 +255,8 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)if (nid <0 ) nid =numa_mem_id (); - error =track_pfn_remap (NULL , &pgprot,PHYS_PFN (res->start),0 , -resource_size (res)); + error =track_pfn_remap (NULL , ¶ms.pgprot,PHYS_PFN (res->start), +0 ,resource_size (res));if (error) goto err_pfn_remap;
Signature · 1 line
2.20.1
PATCH v4 7/7 mm/memremap: Set caching mode for PCI P2PDMA memory to WC
In reply to Logan GunthorpePCI BAR IO memory should never be mapped as WB, however prior to this the PAT bits were set WB and it was typically overridden by MTRR registers set by the firmware.
Set PCI P2PDMA memory to be UC as this is what it currently, typically, ends up being mapped as on x86 after the MTRR registers override the cache setting.
Future use-cases may need to generalize this by adding flags to select the caching type, as some P2PDMA cases may not want UC. However, those use-cases are not upstream yet and this can be changed when they arrive.
--- mm/memremap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/memremap.c +3 -0
@@ -190,7 +190,10 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) }break ;case MEMORY_DEVICE_DEVDAX : + need_devmap_managed = false; +break ;case MEMORY_DEVICE_PCI_P2PDMA : + params.pgprot =pgprot_noncached (params.pgprot); need_devmap_managed = false;break ;default :
Signature · 1 line
2.20.1