revamp vmem_altmap / dev_pagemap handling V3
Subsystems (17)
Hi all,
this series started with two patches from Logan that now are in the middle of the series to kill the memremap-internal pgmap structure and to redo the dev_memreamp_pages interface to be better suitable for future PCI P2P uses. I reviewed them and noticed that there isn't really any good reason to keep struct vmem_altmap either, and that a lot of these alternative device page map access should be better abstracted out instead of being sprinkled all over the mm code. But when we got the RCU warnings in V1 I went for yet another approach, and now struct vmem_altmap is kept for now, but passed explicitly through the memory hotplug code instead of having to do unprotected lookups through the radix tree. The end result is that only the get_user_pages path ever looks up struct dev_pagemap, and struct vmem_altmap is now always embedded into struct dev_pagemap, and explicitly passed where needed.
Please review carefully, this has only been tested with my legacy e820 NVDIMM system.
Chances since V2:
- properly pass altmap from dev_devm_memremap_pages through add_pages (Dan Williams)
- small changelog updates
- a comment type fix
- dropped the patch to just rely on the radix_tree_insert return value
- initialize pgmap->type
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 1/17 memremap: provide stubs for vmem_altmap_offset and vmem_altmap_free
Currently all calls to those functions are eliminated by the compiler when CONFIG_ZONE_DEVICE is not set, but this soon won't be the case.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- include/linux/memremap.h | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
include/linux/memremap.h +14 -4
@@ -26,9 +26,6 @@ struct vmem_altmap {unsigned long alloc; }; -unsigned long vmem_altmap_offset (struct vmem_altmap *altmap); -void vmem_altmap_free (struct vmem_altmap *altmap,unsigned long nr_pfns); -#ifdef CONFIG_ZONE_DEVICE struct vmem_altmap *to_vmem_altmap (unsigned long memmap_start);#else
@@ -138,6 +135,9 @@ void *devm_memremap_pages(struct device *dev, struct resource *res, struct percpu_ref *ref, struct vmem_altmap *altmap);struct dev_pagemap *find_dev_pagemap (resource_size_t phys); +unsigned long vmem_altmap_offset (struct vmem_altmap *altmap); +void vmem_altmap_free (struct vmem_altmap *altmap,unsigned long nr_pfns); +static inline bool is_zone_device_page (const struct page *page);#else static inline void *devm_memremap_pages(struct device *dev,
@@ -157,7 +157,17 @@ static inline struct dev_pagemap *find_dev_pagemap(resource_size_t phys) {return NULL ; } -#endif + +static inline unsigned long vmem_altmap_offset (struct vmem_altmap *altmap) +{ +return 0 ; +} + +static inline void vmem_altmap_free (struct vmem_altmap *altmap, +unsigned long nr_pfns) +{ +} +#endif /* CONFIG_ZONE_DEVICE */ #if defined(CONFIG_DEVICE_PRIVATE ) || defined(CONFIG_DEVICE_PUBLIC )static inline bool is_device_private_page (const struct page *page)
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 2/17 mm: don't export arch_add_memory
In reply to Christoph HellwigOnly x86_64 and sh export this symbol, and it is not used by any modular code.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/sh/mm/init.c | 1 - arch/x86/mm/init_64.c | 1 - 2 files changed, 2 deletions(-)
arch/sh/mm/init.c +0 -1
@@ -498,7 +498,6 @@ int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)return ret; } -EXPORT_SYMBOL_GPL (arch_add_memory);#ifdef CONFIG_NUMA int memory_add_physaddr_to_nid (u64 addr)
arch/x86/mm/init_64.c +0 -1
@@ -796,7 +796,6 @@ int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)return add_pages (nid, start_pfn, nr_pages, want_memblock); } -EXPORT_SYMBOL_GPL (arch_add_memory);#define PAGE_INUSE 0xFD
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 3/17 mm: don't export __add_pages
In reply to Christoph HellwigThis function isn't used by any modules, and is only to be called from core MM code. This includes the calls for the add_pages wrapper that might be inlined.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- mm/memory_hotplug.c | 1 - 1 file changed, 1 deletion(-)
mm/memory_hotplug.c +0 -1
@@ -334,7 +334,6 @@ int __ref __add_pages(int nid, unsigned long phys_start_pfn, out:return err; } -EXPORT_SYMBOL_GPL (__add_pages);#ifdef CONFIG_MEMORY_HOTREMOVE /* find the smallest valid pfn in the range [start_pfn, end_pfn) */
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 4/17 mm: pass the vmem_altmap to arch_add_memory and __add_pages
In reply to Christoph HellwigWe can just pass this on instead of having to do a radix tree lookup without proper locking 2 levels into the callchain.
Signed-off-by Christoph Hellwig <[email protected]>
--- arch/ia64/mm/init.c | 5 +++-- arch/powerpc/mm/mem.c | 5 +++-- arch/s390/mm/init.c | 5 +++-- arch/sh/mm/init.c | 5 +++-- arch/x86/mm/init_32.c | 5 +++-- arch/x86/mm/init_64.c | 11 ++++++----- include/linux/memory_hotplug.h | 17 ++++++++++------- kernel/memremap.c | 3 ++- mm/hmm.c | 5 +++-- mm/memory_hotplug.c | 7 +++---- 10 files changed, 39 insertions(+), 29 deletions(-)
arch/ia64/mm/init.c +3 -2
@@ -647,13 +647,14 @@ mem_init (void) }#ifdef CONFIG_MEMORY_HOTPLUG -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;int ret; - ret =__add_pages (nid, start_pfn, nr_pages, want_memblock); + ret =__add_pages (nid, start_pfn, nr_pages, altmap, want_memblock);if (ret)printk ("%s: Problem encountered in __add_pages() as ret=%d\n" , __func__, ret);
arch/powerpc/mm/mem.c +3 -2
@@ -127,7 +127,8 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)return -ENODEV ; } -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;
@@ -144,7 +145,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)return -EFAULT ; } -return __add_pages (nid, start_pfn, nr_pages, want_memblock); +return __add_pages (nid, start_pfn, nr_pages, altmap, want_memblock); }#ifdef CONFIG_MEMORY_HOTREMOVE
arch/s390/mm/init.c +3 -2
@@ -222,7 +222,8 @@ device_initcall(s390_cma_mem_init);#endif /* CONFIG_CMA */ -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn =PFN_DOWN (start);unsigned long size_pages =PFN_DOWN (size);
@@ -232,7 +233,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)if (rc)return rc; - rc =__add_pages (nid, start_pfn, size_pages, want_memblock); + rc =__add_pages (nid, start_pfn, size_pages, altmap, want_memblock);if (rc)vmem_remove_mapping (start, size);return rc;
arch/sh/mm/init.c +3 -2
@@ -485,14 +485,15 @@ void free_initrd_mem(unsigned long start, unsigned long end)#endif #ifdef CONFIG_MEMORY_HOTPLUG -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn =PFN_DOWN (start);unsigned long nr_pages = size >>PAGE_SHIFT ;int ret;/* We only have ZONE_NORMAL, so this is easy.. */ - ret =__add_pages (nid, start_pfn, nr_pages, want_memblock); + ret =__add_pages (nid, start_pfn, nr_pages, altmap, want_memblock);if (unlikely (ret))printk ("%s: Failed, __add_pages() == %d\n" , __func__, ret);
arch/x86/mm/init_32.c +3 -2
@@ -829,12 +829,13 @@ void __init mem_init(void) }#ifdef CONFIG_MEMORY_HOTPLUG -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; -return __add_pages (nid, start_pfn, nr_pages, want_memblock); +return __add_pages (nid, start_pfn, nr_pages, altmap, want_memblock); }#ifdef CONFIG_MEMORY_HOTREMOVE
arch/x86/mm/init_64.c +6 -5
@@ -772,12 +772,12 @@ static void update_end_of_memory_vars(u64 start, u64 size) } } -int add_pages (int nid,unsigned long start_pfn, -unsigned long nr_pages,bool want_memblock) +int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, +struct vmem_altmap *altmap,bool want_memblock) {int ret; - ret =__add_pages (nid, start_pfn, nr_pages, want_memblock); + ret =__add_pages (nid, start_pfn, nr_pages, altmap, want_memblock);WARN_ON_ONCE (ret);/* update max_pfn, max_low_pfn and high_memory */
@@ -787,14 +787,15 @@ int add_pages(int nid, unsigned long start_pfn,return ret; } -int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock) +int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;init_memory_mapping (start, start + size); -return add_pages (nid, start_pfn, nr_pages, want_memblock); +return add_pages (nid, start_pfn, nr_pages, altmap, want_memblock); }#define PAGE_INUSE 0xFD
include/linux/memory_hotplug.h +10 -7
@@ -13,6 +13,7 @@ struct pglist_data;struct mem_section ;struct memory_block ;struct resource ; +struct vmem_altmap ;#ifdef CONFIG_MEMORY_HOTPLUG /*
@@ -131,18 +132,19 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,#endif /* CONFIG_MEMORY_HOTREMOVE */ /* reasonably generic interface to expand the physical pages */ -extern int __add_pages (int nid,unsigned long start_pfn, -unsigned long nr_pages,bool want_memblock); +extern int __add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, +struct vmem_altmap *altmap,bool want_memblock);#ifndef CONFIG_ARCH_HAS_ADD_PAGES static inline int add_pages (int nid,unsigned long start_pfn, -unsigned long nr_pages,bool want_memblock) +unsigned long nr_pages,struct vmem_altmap *altmap, +bool want_memblock) { -return __add_pages (nid, start_pfn, nr_pages, want_memblock); +return __add_pages (nid, start_pfn, nr_pages, altmap, want_memblock); }#else /* ARCH_HAS_ADD_PAGES */ -int add_pages (int nid,unsigned long start_pfn, -unsigned long nr_pages,bool want_memblock); +int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages, +struct vmem_altmap *altmap,bool want_memblock);#endif /* ARCH_HAS_ADD_PAGES */ #ifdef CONFIG_NUMA
@@ -318,7 +320,8 @@ extern int walk_memory_range(unsigned long start_pfn, unsigned long end_pfn, void *arg, int (*func)(struct memory_block *, void *));extern int add_memory (int nid,u64 start,u64 size);extern int add_memory_resource (int nid,struct resource *resource,bool online); -extern int arch_add_memory (int nid,u64 start,u64 size,bool want_memblock); +extern int arch_add_memory (int nid,u64 start,u64 size, +struct vmem_altmap *altmap,bool want_memblock);extern void move_pfn_range_to_zone (struct zone *zone,unsigned long start_pfn,unsigned long nr_pages);extern int offline_pages (unsigned long start_pfn,unsigned long nr_pages);
kernel/memremap.c +2 -1
@@ -382,6 +382,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,if (altmap) {memcpy (&page_map->altmap, altmap,sizeof (*altmap)); pgmap->altmap = &page_map->altmap; + altmap = pgmap->altmap; } pgmap->ref = ref; pgmap->res = &page_map->res;
@@ -427,7 +428,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res, goto err_pfn_remap;mem_hotplug_begin (); - error =arch_add_memory (nid, align_start, align_size, false); + error =arch_add_memory (nid, align_start, align_size, altmap, false);if (!error) move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], align_start >> PAGE_SHIFT,
mm/hmm.c +3 -2
@@ -931,10 +931,11 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem)* want the linear mapping and thus use arch_add_memory(). */ if (devmem->pagemap.type ==MEMORY_DEVICE_PUBLIC ) - ret =arch_add_memory (nid, align_start, align_size, false); + ret =arch_add_memory (nid, align_start, align_size,NULL , + false);else ret =add_pages (nid, align_start >>PAGE_SHIFT , - align_size >>PAGE_SHIFT , false); + align_size >>PAGE_SHIFT ,NULL , false);if (ret) {mem_hotplug_done (); goto error_add_memory;
mm/memory_hotplug.c +3 -4
@@ -292,18 +292,17 @@ static int __meminit __add_section(int nid, unsigned long phys_start_pfn,* add the new pages. */ int __ref __add_pages (int nid,unsigned long phys_start_pfn, -unsigned long nr_pages,bool want_memblock) +unsigned long nr_pages,struct vmem_altmap *altmap, +bool want_memblock) {unsigned long i;int err =0 ;int start_sec, end_sec; -struct vmem_altmap *altmap;/* during initialize mem_map, align hot-added range to section */ start_sec =pfn_to_section_nr (phys_start_pfn); end_sec =pfn_to_section_nr (phys_start_pfn + nr_pages -1 ); - altmap =to_vmem_altmap ((unsigned long )pfn_to_page (phys_start_pfn));if (altmap) {/* * Validate altmap is within bounds of the total request
@@ -1148,7 +1147,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online) }/* call arch's memory hotadd */ - ret =arch_add_memory (nid, start, size, true); + ret =arch_add_memory (nid, start, size,NULL , true);if (ret <0 ) goto error;
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 5/17 mm: pass the vmem_altmap to vmemmap_populate
In reply to Christoph HellwigWe can just pass this on instead of having to do a radix tree lookup without proper locking a few levels into the callchain.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/arm64/mm/mmu.c | 6 ++++-- arch/ia64/mm/discontig.c | 3 ++- arch/powerpc/mm/init_64.c | 7 ++----- arch/s390/mm/vmem.c | 3 ++- arch/sparc/mm/init_64.c | 2 +- arch/x86/mm/init_64.c | 4 ++-- include/linux/memory_hotplug.h | 3 ++- include/linux/mm.h | 6 ++++-- mm/memory_hotplug.c | 7 ++++--- mm/sparse-vmemmap.c | 7 ++++--- mm/sparse.c | 20 ++++++++++++-------- 11 files changed, 39 insertions(+), 29 deletions(-)
arch/arm64/mm/mmu.c +4 -2
@@ -654,12 +654,14 @@ int kern_addr_valid(unsigned long addr) } #ifdef CONFIG_SPARSEMEM_VMEMMAP #if !ARM64_SWAPPER_USES_SECTION_MAPS -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) { return vmemmap_populate_basepages(start, end, node); } #else /* !ARM64_SWAPPER_USES_SECTION_MAPS */ -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) { unsigned long addr = start; unsigned long next;
arch/ia64/mm/discontig.c +2 -1
@@ -754,7 +754,8 @@ void arch_refresh_nodedata(int update_node, pg_data_t *update_pgdat)#endif #ifdef CONFIG_SPARSEMEM_VMEMMAP -int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node) +int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) {return vmemmap_populate_basepages (start, end, node); }
arch/powerpc/mm/init_64.c +2 -5
@@ -183,7 +183,8 @@ static __meminit void vmemmap_list_populate(unsigned long phys, vmemmap_list = vmem_back; } -int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node) +int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) {unsigned long page_size =1 << mmu_psize_defs[mmu_vmemmap_psize].shift;
@@ -193,16 +194,12 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node)pr_debug ("vmemmap_populate %lx..%lx, node %d\n" , start, end, node);for (; start < end; start += page_size) { -struct vmem_altmap *altmap;void *p;int rc;if (vmemmap_populated (start, page_size))continue ; -/* altmap lookups only work at section boundaries */ - altmap =to_vmem_altmap (SECTION_ALIGN_DOWN (start)); - p =__vmemmap_alloc_block_buf (page_size, node, altmap);if (!p)return -ENOMEM ;
arch/s390/mm/vmem.c +2 -1
@@ -211,7 +211,8 @@ static void vmem_remove_range(unsigned long start, unsigned long size)/* * Add a backed mem_map array to the virtual mem_map array. */ -int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node) +int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) {unsigned long pgt_prot, sgt_prot;unsigned long address = start;
arch/sparc/mm/init_64.c +1 -1
@@ -2628,7 +2628,7 @@ EXPORT_SYMBOL(_PAGE_CACHE);#ifdef CONFIG_SPARSEMEM_VMEMMAP int __meminit vmemmap_populate (unsigned long vstart,unsigned long vend, -int node) +int node,struct vmem_altmap *altmap) {unsigned long pte_base;
arch/x86/mm/init_64.c +2 -2
@@ -1411,9 +1411,9 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,return 0 ; } -int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node) +int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) { -struct vmem_altmap *altmap =to_vmem_altmap (start);int err;if (boot_cpu_has (X86_FEATURE_PSE ))
include/linux/memory_hotplug.h +2 -1
@@ -327,7 +327,8 @@ extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,extern int offline_pages (unsigned long start_pfn,unsigned long nr_pages);extern bool is_memblock_offlined (struct memory_block *mem);extern void remove_memory (int nid,u64 start,u64 size); -extern int sparse_add_one_section (struct pglist_data *pgdat,unsigned long start_pfn); +extern int sparse_add_one_section (struct pglist_data *pgdat, +unsigned long start_pfn,struct vmem_altmap *altmap);extern void sparse_remove_one_section (struct zone *zone,struct mem_section *ms,unsigned long map_offset); extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
include/linux/mm.h +4 -2
@@ -2538,7 +2538,8 @@ void sparse_mem_maps_populate_node(struct page **map_map,unsigned long map_count, int nodeid); -struct page *sparse_mem_map_populate (unsigned long pnum,int nid); +struct page *sparse_mem_map_populate (unsigned long pnum,int nid, +struct vmem_altmap *altmap);pgd_t *vmemmap_pgd_populate (unsigned long addr,int node);p4d_t *vmemmap_p4d_populate (pgd_t *pgd,unsigned long addr,int node);pud_t *vmemmap_pud_populate (p4d_t *p4d,unsigned long addr,int node);
@@ -2556,7 +2557,8 @@ static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)void vmemmap_verify (pte_t *,int ,unsigned long ,unsigned long );int vmemmap_populate_basepages (unsigned long start,unsigned long end,int node); -int vmemmap_populate (unsigned long start,unsigned long end,int node); +int vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap);void vmemmap_populate_print_last (void );#ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free (unsigned long start,unsigned long end);
mm/memory_hotplug.c +4 -3
@@ -250,7 +250,7 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat)#endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */ static int __meminit __add_section (int nid,unsigned long phys_start_pfn, -bool want_memblock) +struct vmem_altmap *altmap,bool want_memblock) {int ret;int i;
@@ -258,7 +258,7 @@ static int __meminit __add_section(int nid, unsigned long phys_start_pfn,if (pfn_valid (phys_start_pfn))return -EEXIST ; - ret =sparse_add_one_section (NODE_DATA (nid), phys_start_pfn); + ret =sparse_add_one_section (NODE_DATA (nid), phys_start_pfn, altmap);if (ret <0 )return ret;
@@ -317,7 +317,8 @@ int __ref __add_pages(int nid, unsigned long phys_start_pfn, }for (i = start_sec; i <= end_sec; i++) { - err =__add_section (nid,section_nr_to_pfn (i), want_memblock); + err =__add_section (nid,section_nr_to_pfn (i), altmap, + want_memblock);/* * EEXIST is finally dealt with by ioresource collision
mm/sparse-vmemmap.c +4 -3
@@ -278,7 +278,8 @@ int __meminit vmemmap_populate_basepages(unsigned long start,return 0 ; } -struct page *__meminit sparse_mem_map_populate (unsigned long pnum,int nid) +struct page *__meminit sparse_mem_map_populate (unsigned long pnum,int nid, +struct vmem_altmap *altmap) {unsigned long start;unsigned long end;
@@ -288,7 +289,7 @@ struct page * __meminit sparse_mem_map_populate(unsigned long pnum, int nid) start = (unsigned long )map; end = (unsigned long )(map +PAGES_PER_SECTION ); -if (vmemmap_populate (start, end, nid)) +if (vmemmap_populate (start, end, nid, altmap))return NULL ;return map;
@@ -318,7 +319,7 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,if (!present_section_nr (pnum))continue ; - map_map[pnum] =sparse_mem_map_populate (pnum, nodeid); + map_map[pnum] =sparse_mem_map_populate (pnum, nodeid,NULL );if (map_map[pnum])continue ; ms =__nr_to_section (pnum);
mm/sparse.c +12 -8
@@ -417,7 +417,8 @@ static void __init sparse_early_usemaps_alloc_node(void *data, }#ifndef CONFIG_SPARSEMEM_VMEMMAP -struct page __init *sparse_mem_map_populate (unsigned long pnum,int nid) +struct page __init *sparse_mem_map_populate (unsigned long pnum,int nid, +struct vmem_altmap *altmap) {struct page *map;unsigned long size;
@@ -472,7 +473,7 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,if (!present_section_nr (pnum))continue ; - map_map[pnum] =sparse_mem_map_populate (pnum, nodeid); + map_map[pnum] =sparse_mem_map_populate (pnum, nodeid,NULL );if (map_map[pnum])continue ; ms =__nr_to_section (pnum);
@@ -500,7 +501,7 @@ static struct page __init *sparse_early_mem_map_alloc(unsigned long pnum)struct mem_section *ms =__nr_to_section (pnum);int nid =sparse_early_nid (ms); - map =sparse_mem_map_populate (pnum, nid); + map =sparse_mem_map_populate (pnum, nid,NULL );if (map)return map;
@@ -678,10 +679,11 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn)#endif #ifdef CONFIG_SPARSEMEM_VMEMMAP -static inline struct page *kmalloc_section_memmap (unsigned long pnum,int nid) +static inline struct page *kmalloc_section_memmap (unsigned long pnum,int nid, +struct vmem_altmap *altmap) {/* This will make the necessary allocations eventually. */ -return sparse_mem_map_populate (pnum, nid); +return sparse_mem_map_populate (pnum, nid, altmap); }static void __kfree_section_memmap (struct page *memmap) {
@@ -721,7 +723,8 @@ static struct page *__kmalloc_section_memmap(void)return ret; } -static inline struct page *kmalloc_section_memmap (unsigned long pnum,int nid) +static inline struct page *kmalloc_section_memmap (unsigned long pnum,int nid, +struct vmem_altmap *altmap) {return __kmalloc_section_memmap (); }
@@ -773,7 +776,8 @@ static void free_map_bootmem(struct page *memmap)* set. If this is <=0, then that means that the passed-in * map was not consumed and must be freed. */ -int __meminit sparse_add_one_section (struct pglist_data *pgdat,unsigned long start_pfn) +int __meminit sparse_add_one_section (struct pglist_data *pgdat, +unsigned long start_pfn,struct vmem_altmap *altmap) {unsigned long section_nr =pfn_to_section_nr (start_pfn);struct mem_section *ms;
@@ -789,7 +793,7 @@ int __meminit sparse_add_one_section(struct pglist_data *pgdat, unsigned long st ret =sparse_index_init (section_nr, pgdat->node_id);if (ret <0 && ret != -EEXIST )return ret; - memmap =kmalloc_section_memmap (section_nr, pgdat->node_id); + memmap =kmalloc_section_memmap (section_nr, pgdat->node_id, altmap);if (!memmap)return -ENOMEM ; usemap =__kmalloc_section_usemap ();
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 6/17 mm: pass the vmem_altmap to arch_remove_memory and __remove_pages
In reply to Christoph HellwigWe can just pass this on instead of having to do a radix tree lookup without proper locking 2 levels into the callchain.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/ia64/mm/init.c | 4 ++-- arch/powerpc/mm/mem.c | 6 ++---- arch/s390/mm/init.c | 2 +- arch/sh/mm/init.c | 4 ++-- arch/x86/mm/init_32.c | 4 ++-- arch/x86/mm/init_64.c | 6 ++---- include/linux/memory_hotplug.h | 5 +++-- kernel/memremap.c | 2 +- mm/hmm.c | 4 ++-- mm/memory_hotplug.c | 8 ++------ 10 files changed, 19 insertions(+), 26 deletions(-)
arch/ia64/mm/init.c +2 -2
@@ -663,7 +663,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, }#ifdef CONFIG_MEMORY_HOTREMOVE -int arch_remove_memory (u64 start,u64 size) +int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;
@@ -671,7 +671,7 @@ int arch_remove_memory(u64 start, u64 size)int ret; zone =page_zone (pfn_to_page (start_pfn)); - ret =__remove_pages (zone, start_pfn, nr_pages); + ret =__remove_pages (zone, start_pfn, nr_pages, altmap);if (ret)pr_warn ("%s: Problem encountered in __remove_pages() as" " ret=%d\n" , __func__, ret);
arch/powerpc/mm/mem.c +2 -4
@@ -149,11 +149,10 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, }#ifdef CONFIG_MEMORY_HOTREMOVE -int arch_remove_memory (u64 start,u64 size) +int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ; -struct vmem_altmap *altmap;struct page *page;int ret;
@@ -162,11 +161,10 @@ int arch_remove_memory(u64 start, u64 size)* when querying the zone. */ page =pfn_to_page (start_pfn); - altmap =to_vmem_altmap ((unsigned long ) page);if (altmap) page +=vmem_altmap_offset (altmap); - ret =__remove_pages (page_zone (page), start_pfn, nr_pages); + ret =__remove_pages (page_zone (page), start_pfn, nr_pages, altmap);if (ret)return ret;
arch/s390/mm/init.c +1 -1
@@ -240,7 +240,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, }#ifdef CONFIG_MEMORY_HOTREMOVE -int arch_remove_memory (u64 start,u64 size) +int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {/* * There is no hardware or firmware interface which could trigger a
arch/sh/mm/init.c +2 -2
@@ -510,7 +510,7 @@ EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);#endif #ifdef CONFIG_MEMORY_HOTREMOVE -int arch_remove_memory (u64 start,u64 size) +int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {unsigned long start_pfn =PFN_DOWN (start);unsigned long nr_pages = size >>PAGE_SHIFT ;
@@ -518,7 +518,7 @@ int arch_remove_memory(u64 start, u64 size)int ret; zone =page_zone (pfn_to_page (start_pfn)); - ret =__remove_pages (zone, start_pfn, nr_pages); + ret =__remove_pages (zone, start_pfn, nr_pages, altmap);if (unlikely (ret))pr_warn ("%s: Failed, __remove_pages() == %d\n" , __func__, ret);
arch/x86/mm/init_32.c +2 -2
@@ -839,14 +839,14 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, }#ifdef CONFIG_MEMORY_HOTREMOVE -int arch_remove_memory (u64 start,u64 size) +int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;struct zone *zone; zone =page_zone (pfn_to_page (start_pfn)); -return __remove_pages (zone, start_pfn, nr_pages); +return __remove_pages (zone, start_pfn, nr_pages, altmap); }#endif #endif
arch/x86/mm/init_64.c +2 -4
@@ -1132,21 +1132,19 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end)remove_pagetable (start, end, true); } -int __ref arch_remove_memory (u64 start,u64 size) +int __ref arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) {unsigned long start_pfn = start >>PAGE_SHIFT ;unsigned long nr_pages = size >>PAGE_SHIFT ;struct page *page =pfn_to_page (start_pfn); -struct vmem_altmap *altmap;struct zone *zone;int ret;/* With altmap the first mapped page is offset from @start */ - altmap =to_vmem_altmap ((unsigned long ) page);if (altmap) page +=vmem_altmap_offset (altmap); zone =page_zone (page); - ret =__remove_pages (zone, start_pfn, nr_pages); + ret =__remove_pages (zone, start_pfn, nr_pages, altmap);WARN_ON_ONCE (ret);kernel_physical_mapping_remove (start, start + size);
include/linux/memory_hotplug.h +3 -2
@@ -126,9 +126,10 @@ static inline bool movable_node_is_enabled(void)#ifdef CONFIG_MEMORY_HOTREMOVE extern bool is_pageblock_removable_nolock (struct page *page); -extern int arch_remove_memory (u64 start,u64 size); +extern int arch_remove_memory (u64 start,u64 size, +struct vmem_altmap *altmap);extern int __remove_pages (struct zone *zone,unsigned long start_pfn, -unsigned long nr_pages); +unsigned long nr_pages,struct vmem_altmap *altmap);#endif /* CONFIG_MEMORY_HOTREMOVE */ /* reasonably generic interface to expand the physical pages */
kernel/memremap.c +1 -1
@@ -304,7 +304,7 @@ static void devm_memremap_pages_release(struct device *dev, void *data) align_size =ALIGN (resource_size (res),SECTION_SIZE );mem_hotplug_begin (); -arch_remove_memory (align_start, align_size); +arch_remove_memory (align_start, align_size, pgmap->altmap);mem_hotplug_done ();untrack_pfn (NULL ,PHYS_PFN (align_start), align_size);
mm/hmm.c +2 -2
@@ -838,10 +838,10 @@ static void hmm_devmem_release(struct device *dev, void *data)mem_hotplug_begin ();if (resource->desc ==IORES_DESC_DEVICE_PRIVATE_MEMORY ) -__remove_pages (zone, start_pfn, npages); +__remove_pages (zone, start_pfn, npages,NULL );else arch_remove_memory (start_pfn <<PAGE_SHIFT , - npages <<PAGE_SHIFT ); + npages <<PAGE_SHIFT ,NULL );mem_hotplug_done ();hmm_devmem_radix_release (resource);
mm/memory_hotplug.c +2 -6
@@ -569,7 +569,7 @@ static int __remove_section(struct zone *zone, struct mem_section *ms,* calling offline_pages(). */ int __remove_pages (struct zone *zone,unsigned long phys_start_pfn, -unsigned long nr_pages) +unsigned long nr_pages,struct vmem_altmap *altmap) {unsigned long i;unsigned long map_offset =0 ;
@@ -577,10 +577,6 @@ int __remove_pages(struct zone *zone, unsigned long phys_start_pfn,/* In the ZONE_DEVICE case device driver owns the memory region */ if (is_dev_zone (zone)) { -struct page *page =pfn_to_page (phys_start_pfn); -struct vmem_altmap *altmap; - - altmap =to_vmem_altmap ((unsigned long ) page);if (altmap) map_offset =vmem_altmap_offset (altmap); }else {
@@ -1890,7 +1886,7 @@ void __ref remove_memory(int nid, u64 start, u64 size)memblock_free (start, size);memblock_remove (start, size); -arch_remove_memory (start, size); +arch_remove_memory (start, size,NULL );try_offline_node (nid);
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 7/17 mm: pass the vmem_altmap to vmemmap_free
In reply to Christoph HellwigWe can just pass this on instead of having to do a radix tree lookup without proper locking a few levels into the callchain.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/arm64/mm/mmu.c | 3 +- arch/ia64/mm/discontig.c | 3 +- arch/powerpc/mm/init_64.c | 5 ++-- arch/s390/mm/vmem.c | 3 +- arch/sparc/mm/init_64.c | 3 +- arch/x86/mm/init_64.c | 67 ++++++++++++++++++++++++------------------ include/linux/memory_hotplug.h | 2 +- include/linux/mm.h | 3 +- mm/memory_hotplug.c | 7 +++-- mm/sparse.c | 23 ++++++++------- 10 files changed, 68 insertions(+), 51 deletions(-)
arch/arm64/mm/mmu.c +2 -1
@@ -696,7 +696,8 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,return 0 ; }#endif /* CONFIG_ARM64_64K_PAGES */ -void vmemmap_free (unsigned long start,unsigned long end) +void vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) { }#endif /* CONFIG_SPARSEMEM_VMEMMAP */
arch/ia64/mm/discontig.c +2 -1
@@ -760,7 +760,8 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return vmemmap_populate_basepages(start, end, node); } -void vmemmap_free (unsigned long start,unsigned long end) +void vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) { }#endif
arch/powerpc/mm/init_64.c +2 -3
@@ -254,7 +254,8 @@ static unsigned long vmemmap_list_free(unsigned long start)return vmem_back->phys; } -void __ref vmemmap_free (unsigned long start,unsigned long end) +void __ref vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) {unsigned long page_size =1 << mmu_psize_defs[mmu_vmemmap_psize].shift;unsigned long page_order =get_order (page_size);
@@ -265,7 +266,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long end)for (; start < end; start += page_size) {unsigned long nr_pages, addr; -struct vmem_altmap *altmap;struct page *section_base;struct page *page;
@@ -285,7 +285,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long end) section_base =pfn_to_page (vmemmap_section_start (start)); nr_pages =1 << page_order; - altmap =to_vmem_altmap ((unsigned long ) section_base);if (altmap) {vmem_altmap_free (altmap, nr_pages); }else if (PageReserved (page)) {
arch/s390/mm/vmem.c +2 -1
@@ -297,7 +297,8 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,return ret; } -void vmemmap_free (unsigned long start,unsigned long end) +void vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) { }
arch/sparc/mm/init_64.c +2 -1
@@ -2671,7 +2671,8 @@ int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend, return 0; } -void vmemmap_free (unsigned long start,unsigned long end) +void vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) { }#endif /* CONFIG_SPARSEMEM_VMEMMAP */
arch/x86/mm/init_64.c +38 -29
218 lines not on this page · raw message
include/linux/memory_hotplug.h +1 -1
@@ -331,7 +331,7 @@ extern void remove_memory(int nid, u64 start, u64 size);extern int sparse_add_one_section (struct pglist_data *pgdat,unsigned long start_pfn,struct vmem_altmap *altmap);extern void sparse_remove_one_section (struct zone *zone,struct mem_section *ms, -unsigned long map_offset); +unsigned long map_offset,struct vmem_altmap *altmap);extern struct page *sparse_decode_mem_map (unsigned long coded_mem_map,unsigned long pnum); extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
include/linux/mm.h +2 -1
@@ -2561,7 +2561,8 @@ int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap);void vmemmap_populate_print_last (void );#ifdef CONFIG_MEMORY_HOTPLUG -void vmemmap_free (unsigned long start,unsigned long end); +void vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap);#endif void register_page_bootmem_memmap (unsigned long section_nr,struct page *map,unsigned long nr_pages);
mm/memory_hotplug.c +4 -3
@@ -536,7 +536,7 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn) }static int __remove_section (struct zone *zone,struct mem_section *ms, -unsigned long map_offset) +unsigned long map_offset,struct vmem_altmap *altmap) {unsigned long start_pfn;int scn_nr;
@@ -553,7 +553,7 @@ static int __remove_section(struct zone *zone, struct mem_section *ms, start_pfn =section_nr_to_pfn ((unsigned long )scn_nr);__remove_zone (zone, start_pfn); -sparse_remove_one_section (zone, ms, map_offset); +sparse_remove_one_section (zone, ms, map_offset, altmap);return 0 ; }
@@ -607,7 +607,8 @@ int __remove_pages(struct zone *zone, unsigned long phys_start_pfn,for (i =0 ; i < sections_to_remove; i++) {unsigned long pfn = phys_start_pfn + i*PAGES_PER_SECTION ; - ret =__remove_section (zone,__pfn_to_section (pfn), map_offset); + ret =__remove_section (zone,__pfn_to_section (pfn), map_offset, + altmap); map_offset =0 ;if (ret)break ;
mm/sparse.c +13 -10
@@ -685,12 +685,13 @@ static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,/* This will make the necessary allocations eventually. */ return sparse_mem_map_populate (pnum, nid, altmap); } -static void __kfree_section_memmap (struct page *memmap) +static void __kfree_section_memmap (struct page *memmap, +struct vmem_altmap *altmap) {unsigned long start = (unsigned long )memmap;unsigned long end = (unsigned long )(memmap +PAGES_PER_SECTION ); -vmemmap_free (start, end); +vmemmap_free (start, end, altmap); }#ifdef CONFIG_MEMORY_HOTREMOVE static void free_map_bootmem (struct page *memmap)
@@ -698,7 +699,7 @@ static void free_map_bootmem(struct page *memmap)unsigned long start = (unsigned long )memmap;unsigned long end = (unsigned long )(memmap +PAGES_PER_SECTION ); - vmemmap_free(start, end); + vmemmap_free(start, end, NULL); }#endif /* CONFIG_MEMORY_HOTREMOVE */ #else
@@ -729,7 +730,8 @@ static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,return __kmalloc_section_memmap (); } -static void __kfree_section_memmap (struct page *memmap) +static void __kfree_section_memmap (struct page *memmap, +struct vmem_altmap *altmap) {if (is_vmalloc_addr (memmap))vfree (memmap);
@@ -798,7 +800,7 @@ int __meminit sparse_add_one_section(struct pglist_data *pgdat,return -ENOMEM ; usemap =__kmalloc_section_usemap ();if (!usemap) { -__kfree_section_memmap (memmap); +__kfree_section_memmap (memmap, altmap);return -ENOMEM ; }
@@ -820,7 +822,7 @@ int __meminit sparse_add_one_section(struct pglist_data *pgdat,pgdat_resize_unlock (pgdat, &flags);if (ret <=0 ) {kfree (usemap); -__kfree_section_memmap (memmap); +__kfree_section_memmap (memmap, altmap); }return ret; }
@@ -847,7 +849,8 @@ static inline void clear_hwpoisoned_pages(struct page *memmap, int nr_pages) }#endif -static void free_section_usemap (struct page *memmap,unsigned long *usemap) +static void free_section_usemap (struct page *memmap,unsigned long *usemap, +struct vmem_altmap *altmap) {struct page *usemap_page;
@@ -861,7 +864,7 @@ static void free_section_usemap(struct page *memmap, unsigned long *usemap)if (PageSlab (usemap_page) ||PageCompound (usemap_page)) {kfree (usemap);if (memmap) -__kfree_section_memmap (memmap); +__kfree_section_memmap (memmap, altmap);return ; }
@@ -875,7 +878,7 @@ static void free_section_usemap(struct page *memmap, unsigned long *usemap) }void sparse_remove_one_section (struct zone *zone,struct mem_section *ms, -unsigned long map_offset) +unsigned long map_offset,struct vmem_altmap *altmap) {struct page *memmap =NULL ;unsigned long *usemap =NULL , flags;
@@ -893,7 +896,7 @@ void sparse_remove_one_section(struct zone *zone, struct mem_section *ms,clear_hwpoisoned_pages (memmap + map_offset,PAGES_PER_SECTION - map_offset); -free_section_usemap (memmap, usemap); +free_section_usemap (memmap, usemap, altmap); }#endif /* CONFIG_MEMORY_HOTREMOVE */ #endif /* CONFIG_MEMORY_HOTPLUG */
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 8/17 mm: pass the vmem_altmap to memmap_init_zone
In reply to Christoph HellwigPass the vmem_altmap two levels down instead of needing a lookup.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/ia64/mm/init.c | 9 +++++---- include/linux/memory_hotplug.h | 2 +- include/linux/mm.h | 4 ++-- kernel/memremap.c | 2 +- mm/hmm.c | 2 +- mm/memory_hotplug.c | 9 +++++---- mm/page_alloc.c | 6 +++--- 7 files changed, 18 insertions(+), 16 deletions(-)
arch/ia64/mm/init.c +5 -4
@@ -501,7 +501,7 @@ virtual_memmap_init(u64 start, u64 end, void *arg)if (map_start < map_end)memmap_init_zone ((unsigned long )(map_end - map_start), args->nid, args->zone,page_to_pfn (map_start), -MEMMAP_EARLY ); +MEMMAP_EARLY ,NULL );return 0 ; }
@@ -509,9 +509,10 @@ void __meminit memmap_init (unsigned long size, int nid, unsigned long zone, unsigned long start_pfn) { -if (!vmem_map) -memmap_init_zone (size, nid, zone, start_pfn,MEMMAP_EARLY ); -else { +if (!vmem_map) { +memmap_init_zone (size, nid, zone, start_pfn,MEMMAP_EARLY , +NULL ); + }else {struct page *start;struct memmap_init_callback_data args;
include/linux/memory_hotplug.h +1 -1
@@ -324,7 +324,7 @@ extern int add_memory_resource(int nid, struct resource *resource, bool online);extern int arch_add_memory (int nid,u64 start,u64 size,struct vmem_altmap *altmap,bool want_memblock);extern void move_pfn_range_to_zone (struct zone *zone,unsigned long start_pfn, -unsigned long nr_pages); +unsigned long nr_pages,struct vmem_altmap *altmap);extern int offline_pages (unsigned long start_pfn,unsigned long nr_pages);extern bool is_memblock_offlined (struct memory_block *mem);extern void remove_memory (int nid,u64 start,u64 size);
include/linux/mm.h +2 -2
@@ -2069,8 +2069,8 @@ static inline void zero_resv_unavail(void) {}#endif extern void set_dma_reserve (unsigned long new_dma_reserve); -extern void memmap_init_zone (unsigned long ,int ,unsigned long , -unsigned long ,enum memmap_context ); +extern void memmap_init_zone (unsigned long ,int ,unsigned long ,unsigned long , +enum memmap_context ,struct vmem_altmap *);extern void setup_per_zone_wmarks (void );extern int __meminit init_per_zone_wmark_min (void );extern void mem_init (void );
kernel/memremap.c +1 -1
@@ -432,7 +432,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,if (!error)move_pfn_range_to_zone (&NODE_DATA (nid)->node_zones[ZONE_DEVICE ], align_start >>PAGE_SHIFT , - align_size >>PAGE_SHIFT ); + align_size >>PAGE_SHIFT , altmap);mem_hotplug_done ();if (error) goto err_add_memory;
mm/hmm.c +1 -1
@@ -942,7 +942,7 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem) }move_pfn_range_to_zone (&NODE_DATA (nid)->node_zones[ZONE_DEVICE ], align_start >>PAGE_SHIFT , - align_size >>PAGE_SHIFT ); + align_size >>PAGE_SHIFT ,NULL );mem_hotplug_done ();for (pfn = devmem->pfn_first; pfn < devmem->pfn_last; pfn++) {
mm/memory_hotplug.c +5 -4
@@ -798,8 +798,8 @@ static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned lon pgdat->node_spanned_pages =max (start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn; } -void __ref move_pfn_range_to_zone (struct zone *zone, -unsigned long start_pfn,unsigned long nr_pages) +void __ref move_pfn_range_to_zone (struct zone *zone,unsigned long start_pfn, +unsigned long nr_pages,struct vmem_altmap *altmap) {struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id;
@@ -824,7 +824,8 @@ void __ref move_pfn_range_to_zone(struct zone *zone,* expects the zone spans the pfn range. All the pages in the range * are reserved so nobody should be touching them so we should be safe */ -memmap_init_zone (nr_pages, nid,zone_idx (zone), start_pfn,MEMMAP_HOTPLUG ); +memmap_init_zone (nr_pages, nid,zone_idx (zone), start_pfn, +MEMMAP_HOTPLUG , altmap);set_zone_contiguous (zone); }
@@ -896,7 +897,7 @@ static struct zone * __meminit move_pfn_range(int online_type, int nid,struct zone *zone; zone =zone_for_pfn_range (online_type, nid, start_pfn, nr_pages); -move_pfn_range_to_zone (zone, start_pfn, nr_pages); +move_pfn_range_to_zone (zone, start_pfn, nr_pages,NULL );return zone; }
mm/page_alloc.c +3 -3
@@ -5314,9 +5314,9 @@ void __ref build_all_zonelists(pg_data_t *pgdat)* done. Non-atomic initialization, single-pass. */ void __meminit memmap_init_zone (unsigned long size,int nid,unsigned long zone, -unsigned long start_pfn,enum memmap_context context) +unsigned long start_pfn,enum memmap_context context, +struct vmem_altmap *altmap) { -struct vmem_altmap *altmap =to_vmem_altmap (__pfn_to_phys (start_pfn));unsigned long end_pfn = start_pfn + size;pg_data_t *pgdat =NODE_DATA (nid);unsigned long pfn;
@@ -5417,7 +5417,7 @@ static void __meminit zone_init_free_lists(struct zone *zone)#ifndef __HAVE_ARCH_MEMMAP_INIT#define memmap_init (size, nid, zone, start_pfn) \ - memmap_init_zone((size), (nid), (zone), (start_pfn), MEMMAP_EARLY) + memmap_init_zone((size), (nid), (zone), (start_pfn), MEMMAP_EARLY, NULL)#endif static int zone_batchsize (struct zone *zone)
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 9/17 mm: split altmap memory map allocation from normal case
In reply to Christoph HellwigNo functional changes, just untangling the call chain and document why the altmap is passed around the hotplug code.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Logan Gunthorpe <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- arch/powerpc/mm/init_64.c | 5 ++++- arch/x86/mm/init_64.c | 5 ++++- include/linux/mm.h | 9 ++------- mm/sparse-vmemmap.c | 15 +++------------ 4 files changed, 13 insertions(+), 21 deletions(-)
arch/powerpc/mm/init_64.c +4 -1
@@ -200,7 +200,10 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,if (vmemmap_populated (start, page_size))continue ; - p =__vmemmap_alloc_block_buf (page_size, node, altmap); +if (altmap) + p =altmap_alloc_block_buf (page_size, altmap); +else + p =vmemmap_alloc_block_buf (page_size, node);if (!p)return -ENOMEM ;
arch/x86/mm/init_64.c +4 -1
@@ -1385,7 +1385,10 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,if (pmd_none (*pmd)) {void *p; - p =__vmemmap_alloc_block_buf (PMD_SIZE , node, altmap); +if (altmap) + p =altmap_alloc_block_buf (PMD_SIZE , altmap); +else + p =vmemmap_alloc_block_buf (PMD_SIZE , node);if (p) {pte_t entry;
include/linux/mm.h +2 -7
@@ -2547,13 +2547,8 @@ pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node);pte_t *vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node);void *vmemmap_alloc_block (unsigned long size,int node);struct vmem_altmap ; -void *__vmemmap_alloc_block_buf (unsigned long size,int node, -struct vmem_altmap *altmap); -static inline void *vmemmap_alloc_block_buf (unsigned long size,int node) -{ -return __vmemmap_alloc_block_buf (size, node,NULL ); -} - +void *vmemmap_alloc_block_buf (unsigned long size,int node); +void *altmap_alloc_block_buf (unsigned long size,struct vmem_altmap *altmap);void vmemmap_verify (pte_t *,int ,unsigned long ,unsigned long );int vmemmap_populate_basepages (unsigned long start,unsigned long end,int node);
mm/sparse-vmemmap.c +3 -12
@@ -74,7 +74,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) }/* need to make sure size is all the same during early stage */ -static void *__meminit alloc_block_buf (unsigned long size,int node) +void *__meminit vmemmap_alloc_block_buf (unsigned long size,int node) {void *ptr;
@@ -129,7 +129,7 @@ static unsigned long __meminit vmem_altmap_alloc(struct vmem_altmap *altmap,return pfn + nr_align; } -static void *__meminit altmap_alloc_block_buf (unsigned long size, +void *__meminit altmap_alloc_block_buf (unsigned long size,struct vmem_altmap *altmap) {unsigned long pfn, nr_pfns;
@@ -153,15 +153,6 @@ static void * __meminit altmap_alloc_block_buf(unsigned long size,return ptr; } -/* need to make sure size is all the same during early stage */ -void *__meminit __vmemmap_alloc_block_buf (unsigned long size,int node, -struct vmem_altmap *altmap) -{ -if (altmap) -return altmap_alloc_block_buf (size, altmap); -return alloc_block_buf (size, node); -} -void __meminit vmemmap_verify (pte_t *pte,int node,unsigned long start,unsigned long end) {
@@ -178,7 +169,7 @@ pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node)pte_t *pte =pte_offset_kernel (pmd, addr);if (pte_none (*pte)) {pte_t entry; -void *p =alloc_block_buf (PAGE_SIZE , node); +void *p =vmemmap_alloc_block_buf (PAGE_SIZE , node);if (!p)return NULL ; entry =pfn_pte (__pa (p) >>PAGE_SHIFT ,PAGE_KERNEL );
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 10/17 mm: merge vmem_altmap_alloc into altmap_alloc_block_buf
In reply to Christoph HellwigThere is no clear separation between the two, so merge them.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Logan Gunthorpe <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- mm/sparse-vmemmap.c | 45 ++++++++++++++++----------------------------- 1 file changed, 16 insertions(+), 29 deletions(-)
mm/sparse-vmemmap.c +16 -29
@@ -107,33 +107,16 @@ static unsigned long __meminit vmem_altmap_nr_free(struct vmem_altmap *altmap) }/** -* vmem_altmap_alloc - allocate pages from the vmem_altmap reservation -* @altmap - reserved page pool for the allocation -* @nr_pfns - size (in pages) of the allocation +* altmap_alloc_block_buf - allocate pages from the device page map +* @altmap: device page map +* @size: size (in bytes) of the allocation * -* Allocations are aligned to the size of the request +* Allocations are aligned to the size of the request. */ -static unsigned long __meminit vmem_altmap_alloc (struct vmem_altmap *altmap, -unsigned long nr_pfns) -{ -unsigned long pfn =vmem_altmap_next_pfn (altmap); -unsigned long nr_align; - - nr_align =1UL <<find_first_bit (&nr_pfns,BITS_PER_LONG ); - nr_align =ALIGN (pfn, nr_align) - pfn; - -if (nr_pfns + nr_align >vmem_altmap_nr_free (altmap)) -return ULONG_MAX ; - altmap->alloc += nr_pfns; - altmap->align += nr_align; -return pfn + nr_align; -} -void *__meminit altmap_alloc_block_buf (unsigned long size,struct vmem_altmap *altmap) { -unsigned long pfn, nr_pfns; -void *ptr; +unsigned long pfn, nr_pfns, nr_align;if (size & ~PAGE_MASK ) { pr_warn_once("%s: allocations must be multiple of PAGE_SIZE (%ld)\n",
@@ -141,16 +124,20 @@ void * __meminit altmap_alloc_block_buf(unsigned long size,return NULL ; } + pfn =vmem_altmap_next_pfn (altmap); nr_pfns = size >>PAGE_SHIFT ; - pfn =vmem_altmap_alloc (altmap, nr_pfns); -if (pfn <ULONG_MAX ) - ptr =__va (__pfn_to_phys (pfn)); -else - ptr =NULL ; + nr_align =1UL <<find_first_bit (&nr_pfns,BITS_PER_LONG ); + nr_align =ALIGN (pfn, nr_align) - pfn; +if (nr_pfns + nr_align >vmem_altmap_nr_free (altmap)) +return NULL ; + + altmap->alloc += nr_pfns; + altmap->align += nr_align; + pfn += nr_align; +pr_debug ("%s: pfn: %#lx alloc: %ld align: %ld nr: %#lx\n" , __func__, pfn, altmap->alloc, altmap->align, nr_pfns); - -return ptr; +return __va (__pfn_to_phys (pfn)); } void __meminit vmemmap_verify(pte_t *pte, int node,
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 11/17 mm: move get_dev_pagemap out of line
In reply to Christoph HellwigThis is a pretty big function, which should be out of line in general, and a no-op stub if CONFIG_ZONE_DEVICЕ is not set.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Logan Gunthorpe <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- include/linux/memremap.h | 39 ++++----------------------------------- kernel/memremap.c | 36 ++++++++++++++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 37 deletions(-)
include/linux/memremap.h +4 -35
@@ -133,7 +133,8 @@ struct dev_pagemap {#ifdef CONFIG_ZONE_DEVICE void *devm_memremap_pages (struct device *dev,struct resource *res,struct percpu_ref *ref,struct vmem_altmap *altmap); -struct dev_pagemap *find_dev_pagemap (resource_size_t phys); +struct dev_pagemap *get_dev_pagemap (unsigned long pfn, +struct dev_pagemap *pgmap);unsigned long vmem_altmap_offset (struct vmem_altmap *altmap);void vmem_altmap_free (struct vmem_altmap *altmap,unsigned long nr_pfns);
@@ -153,7 +154,8 @@ static inline void *devm_memremap_pages(struct device *dev,return ERR_PTR (-ENXIO ); } -static inline struct dev_pagemap *find_dev_pagemap (resource_size_t phys) +static inline struct dev_pagemap *get_dev_pagemap (unsigned long pfn, +struct dev_pagemap *pgmap) {return NULL ; }
@@ -183,39 +185,6 @@ static inline bool is_device_public_page(const struct page *page) }#endif /* CONFIG_DEVICE_PRIVATE || CONFIG_DEVICE_PUBLIC */ -/** -* get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn -* @pfn: page frame number to lookup page_map -* @pgmap: optional known pgmap that already has a reference -* -* @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the -* same mapping. -*/ -static inline struct dev_pagemap *get_dev_pagemap (unsigned long pfn, -struct dev_pagemap *pgmap) -{ -const struct resource *res = pgmap ? pgmap->res :NULL ; -resource_size_t phys =PFN_PHYS (pfn); - -/* -* In the cached case we're already holding a live reference so -* we can simply do a blind increment -*/ -if (res && phys >= res->start && phys <= res->end) { -percpu_ref_get (pgmap->ref); -return pgmap; - } - -/* fall back to slow path lookup */ -rcu_read_lock (); - pgmap =find_dev_pagemap (phys); -if (pgmap && !percpu_ref_tryget_live (pgmap->ref)) - pgmap =NULL ; -rcu_read_unlock (); - -return pgmap; -} -static inline void put_dev_pagemap (struct dev_pagemap *pgmap) {if (pgmap)
kernel/memremap.c +34 -2
@@ -314,7 +314,7 @@ static void devm_memremap_pages_release(struct device *dev, void *data) }/* assumes rcu_read_lock() held at entry */ -struct dev_pagemap *find_dev_pagemap (resource_size_t phys) +static struct dev_pagemap *find_dev_pagemap (resource_size_t phys) {struct page_map *page_map;
@@ -501,8 +501,40 @@ struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start) return pgmap ? pgmap->altmap : NULL; } -#endif /* CONFIG_ZONE_DEVICE */ +/** +* get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn +* @pfn: page frame number to lookup page_map +* @pgmap: optional known pgmap that already has a reference +* +* @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the +* same mapping. +*/ +struct dev_pagemap *get_dev_pagemap (unsigned long pfn, +struct dev_pagemap *pgmap) +{ +const struct resource *res = pgmap ? pgmap->res :NULL ; +resource_size_t phys =PFN_PHYS (pfn); + +/* +* In the cached case we're already holding a live reference so +* we can simply do a blind increment +*/ +if (res && phys >= res->start && phys <= res->end) { +percpu_ref_get (pgmap->ref); +return pgmap; + } + +/* fall back to slow path lookup */ +rcu_read_lock (); + pgmap =find_dev_pagemap (phys); +if (pgmap && !percpu_ref_tryget_live (pgmap->ref)) + pgmap =NULL ; +rcu_read_unlock (); + +return pgmap; +} +#endif /* CONFIG_ZONE_DEVICE */ #if IS_ENABLED (CONFIG_DEVICE_PRIVATE ) ||IS_ENABLED (CONFIG_DEVICE_PUBLIC )void put_zone_device_private_or_public_page (struct page *page)
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 12/17 mm: optimize dev_pagemap reference counting around get_dev_pagemap
In reply to Christoph HellwigChange the calling convention so that get_dev_pagemap always consumes the previous reference instead of doing this using an explicit earlier call to put_dev_pagemap in the callers.
The callers will still need to put the final reference after finishing the loop over the pages.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Logan Gunthorpe <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- kernel/memremap.c | 17 +++++++++-------- mm/gup.c | 7 +++++-- 2 files changed, 14 insertions(+), 10 deletions(-)
kernel/memremap.c +9 -8
@@ -507,22 +507,23 @@ struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)* @pfn: page frame number to lookup page_map * @pgmap: optional known pgmap that already has a reference * -* @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the -* same mapping. +* If @pgmap is non-NULL and covers @pfn it will be returned as-is. If @pgmap +* is non-NULL but does not cover @pfn the reference to it will be released. */ struct dev_pagemap *get_dev_pagemap (unsigned long pfn,struct dev_pagemap *pgmap) { -const struct resource *res = pgmap ? pgmap->res :NULL ;resource_size_t phys =PFN_PHYS (pfn);/* -* In the cached case we're already holding a live reference so -* we can simply do a blind increment +* In the cached case we're already holding a live reference. */ -if (res && phys >= res->start && phys <= res->end) { -percpu_ref_get (pgmap->ref); -return pgmap; +if (pgmap) { +const struct resource *res = pgmap ? pgmap->res :NULL ; + +if (res && phys >= res->start && phys <= res->end) +return pgmap; +put_dev_pagemap (pgmap); }/* fall back to slow path lookup */
mm/gup.c +5 -2
@@ -1410,7 +1410,6 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,VM_BUG_ON_PAGE (compound_head (page) != head, page); -put_dev_pagemap (pgmap);SetPageReferenced (page); pages[*nr] = page; (*nr)++;
@@ -1420,6 +1419,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, ret =1 ; pte_unmap: +if (pgmap) +put_dev_pagemap (pgmap);pte_unmap (ptem);return ret; }
@@ -1459,10 +1460,12 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr,SetPageReferenced (page); pages[*nr] = page;get_page (page); -put_dev_pagemap (pgmap); (*nr)++; pfn++; }while (addr +=PAGE_SIZE , addr != end); + +if (pgmap) +put_dev_pagemap (pgmap);return 1 ; }
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 13/17 memremap: remove to_vmem_altmap
In reply to Christoph HellwigAll callers are gone now.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- include/linux/memremap.h | 9 --------- kernel/memremap.c | 26 -------------------------- 2 files changed, 35 deletions(-)
include/linux/memremap.h +0 -9
@@ -26,15 +26,6 @@ struct vmem_altmap {unsigned long alloc; }; -#ifdef CONFIG_ZONE_DEVICE -struct vmem_altmap *to_vmem_altmap (unsigned long memmap_start); -#else -static inline struct vmem_altmap *to_vmem_altmap (unsigned long memmap_start) -{ -return NULL ; -} -#endif -/* * Specialize ZONE_DEVICE memory into multiple types each having differents * usage.
kernel/memremap.c +0 -26
@@ -476,32 +476,6 @@ void vmem_altmap_free(struct vmem_altmap *altmap, unsigned long nr_pfns) altmap->alloc -= nr_pfns; } -struct vmem_altmap *to_vmem_altmap (unsigned long memmap_start) -{ -/* -* 'memmap_start' is the virtual address for the first "struct -* page" in this range of the vmemmap array. In the case of -* CONFIG_SPARSEMEM_VMEMMAP a page_to_pfn conversion is simple -* pointer arithmetic, so we can perform this to_vmem_altmap() -* conversion without concern for the initialization state of -* the struct page fields. -*/ -struct page *page = (struct page *) memmap_start; -struct dev_pagemap *pgmap; - -/* -* Unconditionally retrieve a dev_pagemap associated with the -* given physical address, this is only for use in the -* arch_{add|remove}_memory() for setting up and tearing down -* the memmap. -*/ -rcu_read_lock (); - pgmap =find_dev_pagemap (__pfn_to_phys (page_to_pfn (page))); -rcu_read_unlock (); - -return pgmap ? pgmap->altmap :NULL ; -} -/** * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn * @pfn: page frame number to lookup page_map
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 14/17 memremap: simplify duplicate region handling in devm_memremap_pages
In reply to Christoph Hellwig__radix_tree_insert already checks for duplicates and returns -EEXIST in that case, so remove the duplicate (and racy) duplicates check.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Logan Gunthorpe <[email protected]>
--- kernel/memremap.c | 11 ----------- 1 file changed, 11 deletions(-)
kernel/memremap.c +0 -11
@@ -396,17 +396,6 @@ void *devm_memremap_pages(struct device *dev, struct resource *res, align_end = align_start + align_size -1 ; foreach_order_pgoff(res, order, pgoff) { -struct dev_pagemap *dup; - -rcu_read_lock (); - dup =find_dev_pagemap (res->start +PFN_PHYS (pgoff)); -rcu_read_unlock (); -if (dup) { -dev_err (dev,"%s: %pr collides with mapping for %s\n" , - __func__, res,dev_name (dup->dev)); - error = -EBUSY ; -break ; - } error =__radix_tree_insert (&pgmap_radix,PHYS_PFN (res->start) + pgoff, order, page_map);if (error) {
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 15/17 memremap: drop private struct page_map
In reply to Christoph HellwigFrom: Logan Gunthorpe <[email protected]>
'struct page_map' is a private structure of 'struct dev_pagemap' but the latter replicates all the same fields as the former so there isn't much value in it. Thus drop it in favour of a completely public struct.
This is a clean up in preperation for a more generally useful 'devm_memeremap_pages' interface.
Signed-off-by Logan Gunthorpe <[email protected]>
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- include/linux/memremap.h | 5 ++-- kernel/memremap.c | 68 ++++++++++++++++++------------------------------ mm/hmm.c | 2 +- 3 files changed, 30 insertions(+), 45 deletions(-)
include/linux/memremap.h +3 -2
@@ -113,8 +113,9 @@ typedef void (*dev_page_free_t)(struct page *page, void *data);struct dev_pagemap {dev_page_fault_t page_fault;dev_page_free_t page_free; -struct vmem_altmap *altmap; -const struct resource *res; +struct vmem_altmap altmap; +bool altmap_valid; +struct resource res;struct percpu_ref *ref;struct device *dev;void *data;
kernel/memremap.c +26 -42
164 lines not on this page · raw message
mm/hmm.c +1 -1
@@ -882,7 +882,7 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem)else devmem->pagemap.type =MEMORY_DEVICE_PRIVATE ; - devmem->pagemap.res = devmem->resource; + devmem->pagemap.res = *devmem->resource; devmem->pagemap.page_fault = hmm_devmem_fault; devmem->pagemap.page_free = hmm_devmem_free; devmem->pagemap.dev = devmem->device;
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 16/17 memremap: change devm_memremap_pages interface to use struct dev_pagemap
In reply to Christoph HellwigThis new interface is similar to how struct device (and many others) work. The caller initializes a 'struct dev_pagemap' as required and calls 'devm_memremap_pages'. This allows the pagemap structure to be embedded in another structure and thus container_of can be used. In this way application specific members can be stored in a containing struct.
This will be used by the P2P infrastructure and HMM could probably be cleaned up to use it as well (instead of having it's own, similar 'hmm_devmem_pages_create' function).
Signed-off-by Logan Gunthorpe <[email protected]>
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- drivers/dax/pmem.c | 20 ++++++++------- drivers/nvdimm/nd.h | 9 +++---- drivers/nvdimm/pfn_devs.c | 27 ++++++++++++--------- drivers/nvdimm/pmem.c | 37 +++++++++++++++------------- drivers/nvdimm/pmem.h | 1 + include/linux/memremap.h | 6 ++--- kernel/memremap.c | 51 ++++++++++++++++----------------------- tools/testing/nvdimm/test/iomap.c | 7 +++--- 8 files changed, 77 insertions(+), 81 deletions(-)
drivers/dax/pmem.c +11 -9
@@ -21,6 +21,7 @@struct dax_pmem {struct device *dev;struct percpu_ref ref; +struct dev_pagemap pgmap;struct completion cmp; };
@@ -69,20 +70,23 @@ static int dax_pmem_probe(struct device *dev)struct nd_namespace_common *ndns;struct nd_dax *nd_dax =to_nd_dax (dev);struct nd_pfn *nd_pfn = &nd_dax->nd_pfn; -struct vmem_altmap __altmap, *altmap =NULL ; ndns =nvdimm_namespace_common_probe (dev);if (IS_ERR (ndns))return PTR_ERR (ndns); nsio =to_nd_namespace_io (&ndns->dev); + dax_pmem =devm_kzalloc (dev,sizeof (*dax_pmem),GFP_KERNEL ); +if (!dax_pmem) +return -ENOMEM ; +/* parse the 'pfn' info block via ->rw_bytes */ rc =devm_nsio_enable (dev, nsio);if (rc)return rc; - altmap =nvdimm_setup_pfn (nd_pfn, &res, &__altmap); -if (IS_ERR (altmap)) -return PTR_ERR (altmap); + rc =nvdimm_setup_pfn (nd_pfn, &dax_pmem->pgmap); +if (rc) +return rc;devm_nsio_disable (dev, nsio); pfn_sb = nd_pfn->pfn_sb;
@@ -94,10 +98,6 @@ static int dax_pmem_probe(struct device *dev)return -EBUSY ; } - dax_pmem =devm_kzalloc (dev,sizeof (*dax_pmem),GFP_KERNEL ); -if (!dax_pmem) -return -ENOMEM ; - dax_pmem->dev = dev;init_completion (&dax_pmem->cmp); rc = percpu_ref_init(&dax_pmem->ref, dax_pmem_percpu_release, 0,
@@ -110,7 +110,8 @@ static int dax_pmem_probe(struct device *dev)if (rc)return rc; - addr =devm_memremap_pages (dev, &res, &dax_pmem->ref, altmap); + dax_pmem->pgmap.ref = &dax_pmem->ref; + addr =devm_memremap_pages (dev, &dax_pmem->pgmap);if (IS_ERR (addr))return PTR_ERR (addr);
@@ -120,6 +121,7 @@ static int dax_pmem_probe(struct device *dev)return rc;/* adjust the dax_region resource to the start of data */ +memcpy (&res, &dax_pmem->pgmap.res,sizeof (res)); res.start +=le64_to_cpu (pfn_sb->dataoff); rc =sscanf (dev_name (&ndns->dev),"namespace%d.%d" , ®ion_id, &id);
drivers/nvdimm/nd.h +4 -5
@@ -368,15 +368,14 @@ unsigned int pmem_sector_size(struct nd_namespace_common *ndns);void nvdimm_badblocks_populate (struct nd_region *nd_region,struct badblocks *bb,const struct resource *res);#if IS_ENABLED (CONFIG_ND_CLAIM ) -struct vmem_altmap *nvdimm_setup_pfn (struct nd_pfn *nd_pfn, -struct resource *res,struct vmem_altmap *altmap); +int nvdimm_setup_pfn (struct nd_pfn *nd_pfn,struct dev_pagemap *pgmap);int devm_nsio_enable (struct device *dev,struct nd_namespace_io *nsio);void devm_nsio_disable (struct device *dev,struct nd_namespace_io *nsio);#else -static inline struct vmem_altmap *nvdimm_setup_pfn (struct nd_pfn *nd_pfn, -struct resource *res,struct vmem_altmap *altmap) +static inline int nvdimm_setup_pfn (struct nd_pfn *nd_pfn, +struct dev_pagemap *pgmap) { -return ERR_PTR (-ENXIO ); +return -ENXIO ; }static inline int devm_nsio_enable (struct device *dev,struct nd_namespace_io *nsio)
drivers/nvdimm/pfn_devs.c +15 -12
@@ -542,9 +542,10 @@ static unsigned long init_altmap_reserve(resource_size_t base)return reserve; } -static struct vmem_altmap *__nvdimm_setup_pfn (struct nd_pfn *nd_pfn, -struct resource *res,struct vmem_altmap *altmap) +static int __nvdimm_setup_pfn (struct nd_pfn *nd_pfn,struct dev_pagemap *pgmap) { +struct resource *res = &pgmap->res; +struct vmem_altmap *altmap = &pgmap->altmap;struct nd_pfn_sb *pfn_sb = nd_pfn->pfn_sb;u64 offset =le64_to_cpu (pfn_sb->dataoff);u32 start_pad =__le32_to_cpu (pfn_sb->start_pad);
@@ -561,11 +562,13 @@ static struct vmem_altmap *__nvdimm_setup_pfn(struct nd_pfn *nd_pfn, res->start += start_pad; res->end -= end_trunc; + pgmap->type =MEMORY_DEVICE_HOST ; +if (nd_pfn->mode ==PFN_MODE_RAM ) {if (offset <SZ_8K ) -return ERR_PTR (-EINVAL ); +return -EINVAL ; nd_pfn->npfns =le64_to_cpu (pfn_sb->npfns); - altmap =NULL ; + pgmap->altmap_valid = false; }else if (nd_pfn->mode ==PFN_MODE_PMEM ) { nd_pfn->npfns =PFN_SECTION_ALIGN_UP ((resource_size (res) - offset) /PAGE_SIZE );
@@ -577,10 +580,11 @@ static struct vmem_altmap *__nvdimm_setup_pfn(struct nd_pfn *nd_pfn,memcpy (altmap, &__altmap,sizeof (*altmap)); altmap->free =PHYS_PFN (offset -SZ_8K ); altmap->alloc =0 ; + pgmap->altmap_valid = true; }else -return ERR_PTR (-ENXIO ); +return -ENXIO ; -return altmap; +return 0 ; }static u64 phys_pmem_align_down (struct nd_pfn *nd_pfn,u64 phys)
@@ -708,19 +712,18 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn)* Determine the effective resource range and vmem_altmap from an nd_pfn * instance. */ -struct vmem_altmap *nvdimm_setup_pfn (struct nd_pfn *nd_pfn, -struct resource *res,struct vmem_altmap *altmap) +int nvdimm_setup_pfn (struct nd_pfn *nd_pfn,struct dev_pagemap *pgmap) {int rc;if (!nd_pfn->uuid || !nd_pfn->ndns) -return ERR_PTR (-ENODEV ); +return -ENODEV ; rc =nd_pfn_init (nd_pfn);if (rc) -return ERR_PTR (rc); +return rc; -/* we need a valid pfn_sb before we can init a vmem_altmap */ -return __nvdimm_setup_pfn (nd_pfn, res, altmap); +/* we need a valid pfn_sb before we can init a dev_pagemap */ +return __nvdimm_setup_pfn (nd_pfn, pgmap); }EXPORT_SYMBOL_GPL (nvdimm_setup_pfn);
drivers/nvdimm/pmem.c +20 -17
@@ -298,34 +298,34 @@ static int pmem_attach_disk(struct device *dev, {struct nd_namespace_io *nsio =to_nd_namespace_io (&ndns->dev);struct nd_region *nd_region =to_nd_region (dev->parent); -struct vmem_altmap __altmap, *altmap =NULL ;int nid =dev_to_node (dev), fua, wbc;struct resource *res = &nsio->res; +struct resource bb_res;struct nd_pfn *nd_pfn =NULL ;struct dax_device *dax_dev;struct nd_pfn_sb *pfn_sb;struct pmem_device *pmem; -struct resource pfn_res;struct request_queue *q;struct device *gendev;struct gendisk *disk;void *addr; +int rc; + + pmem =devm_kzalloc (dev,sizeof (*pmem),GFP_KERNEL ); +if (!pmem) +return -ENOMEM ;/* while nsio_rw_bytes is active, parse a pfn info block if present */ if (is_nd_pfn (dev)) { nd_pfn =to_nd_pfn (dev); - altmap =nvdimm_setup_pfn (nd_pfn, &pfn_res, &__altmap); -if (IS_ERR (altmap)) -return PTR_ERR (altmap); + rc =nvdimm_setup_pfn (nd_pfn, &pmem->pgmap); +if (rc) +return rc; }/* we're attaching a block device, disable raw namespace access */ devm_nsio_disable (dev, nsio); - pmem =devm_kzalloc (dev,sizeof (*pmem),GFP_KERNEL ); -if (!pmem) -return -ENOMEM ; -dev_set_drvdata (dev, pmem); pmem->phys_addr = res->start; pmem->size =resource_size (res);
@@ -350,19 +350,22 @@ static int pmem_attach_disk(struct device *dev,return -ENOMEM ; pmem->pfn_flags =PFN_DEV ; + pmem->pgmap.ref = &q->q_usage_counter;if (is_nd_pfn (dev)) { - addr =devm_memremap_pages (dev, &pfn_res, &q->q_usage_counter, - altmap); + addr =devm_memremap_pages (dev, &pmem->pgmap); pfn_sb = nd_pfn->pfn_sb; pmem->data_offset =le64_to_cpu (pfn_sb->dataoff); - pmem->pfn_pad =resource_size (res) -resource_size (&pfn_res); + pmem->pfn_pad =resource_size (res) - +resource_size (&pmem->pgmap.res); pmem->pfn_flags |=PFN_MAP ; - res = &pfn_res;/* for badblocks populate */ - res->start += pmem->data_offset; +memcpy (&bb_res, &pmem->pgmap.res,sizeof (bb_res)); + bb_res.start += pmem->data_offset; }else if (pmem_should_map_pages (dev)) { - addr =devm_memremap_pages (dev, &nsio->res, - &q->q_usage_counter,NULL ); +memcpy (&pmem->pgmap.res, &nsio->res,sizeof (pmem->pgmap.res)); + pmem->pgmap.altmap_valid = false; + addr =devm_memremap_pages (dev, &pmem->pgmap); pmem->pfn_flags |=PFN_MAP ; +memcpy (&bb_res, &pmem->pgmap.res,sizeof (bb_res)); }else addr =devm_memremap (dev, pmem->phys_addr, pmem->size,ARCH_MEMREMAP_PMEM );
@@ -401,7 +404,7 @@ static int pmem_attach_disk(struct device *dev, / 512);if (devm_init_badblocks (dev, &pmem->bb))return -ENOMEM ; -nvdimm_badblocks_populate (nd_region, &pmem->bb, res); +nvdimm_badblocks_populate (nd_region, &pmem->bb, &bb_res); disk->bb = &pmem->bb; dax_dev =alloc_dax (pmem, disk->disk_name, &pmem_dax_ops);
drivers/nvdimm/pmem.h +1 -0
@@ -22,6 +22,7 @@ struct pmem_device {struct badblocks bb;struct dax_device *dax_dev;struct gendisk *disk; +struct dev_pagemap pgmap; }; long __pmem_direct_access(struct pmem_device *pmem, pgoff_t pgoff,
include/linux/memremap.h +2 -4
@@ -123,8 +123,7 @@ struct dev_pagemap { };#ifdef CONFIG_ZONE_DEVICE -void *devm_memremap_pages (struct device *dev,struct resource *res, -struct percpu_ref *ref,struct vmem_altmap *altmap); +void *devm_memremap_pages (struct device *dev,struct dev_pagemap *pgmap);struct dev_pagemap *get_dev_pagemap (unsigned long pfn,struct dev_pagemap *pgmap);
@@ -134,8 +133,7 @@ void vmem_altmap_free(struct vmem_altmap *altmap, unsigned long nr_pfns);static inline bool is_zone_device_page (const struct page *page);#else static inline void *devm_memremap_pages (struct device *dev, -struct resource *res,struct percpu_ref *ref, -struct vmem_altmap *altmap) +struct dev_pagemap *pgmap) {/* * Fail attempts to call devm_memremap_pages() without
kernel/memremap.c +21 -30
97 lines not on this page · raw message
tools/testing/nvdimm/test/iomap.c +3 -4
@@ -104,15 +104,14 @@ void *__wrap_devm_memremap(struct device *dev, resource_size_t offset, }EXPORT_SYMBOL (__wrap_devm_memremap); -void *__wrap_devm_memremap_pages (struct device *dev,struct resource *res, -struct percpu_ref *ref,struct vmem_altmap *altmap) +void *__wrap_devm_memremap_pages (struct device *dev,struct dev_pagemap *pgmap) { -resource_size_t offset = res->start; +resource_size_t offset = pgmap->res.start;struct nfit_test_resource *nfit_res =get_nfit_res (offset);if (nfit_res)return nfit_res->buf + offset - nfit_res->res.start; -return devm_memremap_pages (dev, res, ref, altmap); +return devm_memremap_pages (dev, pgmap); }EXPORT_SYMBOL (__wrap_devm_memremap_pages);
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 17/17 memremap: merge find_dev_pagemap into get_dev_pagemap
In reply to Christoph HellwigThere is only one caller of the trivial function find_dev_pagemap left, so just merge it into the caller.
Signed-off-by Christoph Hellwig <[email protected]>
Reviewed-by Dan Williams <[email protected]>
--- kernel/memremap.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
kernel/memremap.c +1 -9
@@ -306,14 +306,6 @@ static void devm_memremap_pages_release(void *data) "%s: failed to free all reserved pages\n", __func__); } -/* assumes rcu_read_lock() held at entry */ -static struct dev_pagemap *find_dev_pagemap (resource_size_t phys) -{ -WARN_ON_ONCE (!rcu_read_lock_held ()); - -return radix_tree_lookup (&pgmap_radix,PHYS_PFN (phys)); -} -/** * devm_memremap_pages - remap and provide memmap backing for the given resource * @dev: hosting device for @res
@@ -466,7 +458,7 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn,/* fall back to slow path lookup */ rcu_read_lock (); - pgmap =find_dev_pagemap (phys); + pgmap =radix_tree_lookup (&pgmap_radix,PHYS_PFN (phys));if (pgmap && !percpu_ref_tryget_live (pgmap->ref)) pgmap =NULL ;rcu_read_unlock ();
Signature · 6 lines
2.14.2 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Any chance to get this fully reviewed and picked up before the end of the merge window?
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
I'm fine carrying these through the nvdimm tree, but I'd need an ack from the mm folks for all the code touches related to arch_add_memory.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Dan Williams
Quoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
I'm fine carrying these through the nvdimm tree, but I'd need an ack from the mm folks for all the code touches related to
arch_add_memory.
I am sorry to be slow here but I am out of time right now - yeah having a lot of fun kaiser time. I didn't get to look at these patches at all yet but the changelog suggests that you want to remove vmem_altmap. I've had plans to (ab)use this for self hosted struct pages for memory hotplug http://lkml.kernel.org/r/[email protected]
That work is stalled though because it is buggy and I was too busy to finish that work. Anyway, if you believe that removing vmem_altmap is a good step in general I will find another way. I wasn't really happy how the whole thing is grafted to the memory hotplug and (ab)used it only because it was handy and ready for reuse.
Anyway if you need a review of mm parts from me, you will have to wait some more. If this requires some priority then go ahead and merge it. Times are just too crazy right now.
Sorry about that.
Signature · 6 lines
Michal Hocko SUSE Labs _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 8 lines · Michal Hocko
Quoted 6 lines · Dan Williams
Quoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
I'm fine carrying these through the nvdimm tree, but I'd need an ack from the mm folks for all the code touches related to
arch_add_memory.
Quoted 10 lines
I am sorry to be slow here but I am out of time right now - yeah having a lot of fun kaiser time. I didn't get to look at these patches at all yet but the changelog suggests that you want to remove
vmem_altmap. I've had plans to (ab)use this for self hosted struct pages for memory hotplug http://lkml.kernel.org/r/[email protected]
That work is stalled though because it is buggy and I was too busy to finish that work. Anyway, if you believe that removingvmem_altmapis a good step in general I will find another way. I wasn't really happy how the whole thing is grafted to the memory hotplug and (ab)used it only because it was handy and ready for reuse.
You misread, these are keeping vmem_altmap and cleaning up the usage to pass the vmem_altmap pointer through all paths rather than the tricky radix lookup we were doing previously.
Quoted 3 lines
Anyway if you need a review of mm parts from me, you will have to wait some more. If this requires some priority then go ahead and merge it. Times are just too crazy right now.
Since you were planning on reusing vmem_altmap I think these patches make your job easier. I don't see the risk in merging these, we've squeezed out a few bugs and all the nvdimm unit tests are passing.
Quoted 1 line
Sorry about that.
No worries, quite a few of us are in that same boat.
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 20 lines · Dan Williams
Quoted 8 lines · Michal Hocko
Quoted 6 lines · Dan Williams
Quoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
I'm fine carrying these through the nvdimm tree, but I'd need an ack from the mm folks for all the code touches related to
arch_add_memory.Quoted 10 lines
I am sorry to be slow here but I am out of time right now - yeah having a lot of fun kaiser time. I didn't get to look at these patches at all yet but the changelog suggests that you want to remove
vmem_altmap. I've had plans to (ab)use this for self hosted struct pages for memory hotplug http://lkml.kernel.org/r/[email protected]
That work is stalled though because it is buggy and I was too busy to finish that work. Anyway, if you believe that removingvmem_altmapis a good step in general I will find another way. I wasn't really happy how the whole thing is grafted to the memory hotplug and (ab)used it only because it was handy and ready for reuse.
Quoted 3 lines
You misread, these are keeping
vmem_altmapand cleaning up the usage to pass thevmem_altmappointer through all paths rather than the tricky radix lookup we were doing previously.
Good to hear. I really didn't get further than reading through email subjects and for some reason I misread those.
Quoted 7 lines
Quoted 3 lines
Anyway if you need a review of mm parts from me, you will have to wait some more. If this requires some priority then go ahead and merge it. Times are just too crazy right now.
Since you were planning on reusing
vmem_altmapI think these patches make your job easier. I don't see the risk in merging these, we've squeezed out a few bugs and all the nvdimm unit tests are passing.
Good, then really do not wait for me if this aims to get merged soon.
Quoted 3 lines
Quoted 1 line
Sorry about that.
No worries, quite a few of us are in that same boat.
Yeah the boat is quite large I suspect...
Signature · 6 lines
Michal Hocko SUSE Labs _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 6 lines · Dan Williams
Quoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
I'm fine carrying these through the nvdimm tree, but I'd need an ack from the mm folks for all the code touches related to
arch_add_memory.
Looks like we got some semi ACKs, so I'd really like to see this go in now as we have both nvdimm and p2p patches depending on it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jerome Glisse
Reviewed-byQuoted 2 lines · Christoph Hellwig
Any chance to get this fully reviewed and picked up before the end of the merge window?
Sorry for taking so long to get to that, i looked at all the patches and did not see anything obviously wrong and i like the cleanup so
Reviewed-by Jérôme Glisse <[email protected]>
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm