Skip to contentSkip to conversation

add a not device managed memremap_pages v3

PATCH In review Christoph Hellwig 4 participants 19 messages Aug 18, 2019 – Aug 22, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi Dan and Jason,

Bharata has been working on secure page management for kvmppc guests, and one I thing I noticed is that he had to fake up a struct device just so that it could be passed to the devm_memremap_pages instrastructure for device private memory.

This series adds non-device managed versions of the devm_request_free_mem_region and devm_memremap_pages functions for his use case.

Changes since v2:
- improved changelogs that the the v2 changes into account

Changes since v1:
- don't overload devm_request_free_mem_region
- export the memremap_pages and munmap_pages as kvmppc can be a module

#1 lore raw

PATCH 1/4 resource: add a not device managed request_free_mem_region variant

Factor out the guts of devm_request_free_mem_region so that we can implement both a device managed and a manually release version as tiny wrappers around it.

---
 include/linux/ioport.h |  2 ++
 kernel/resource.c      | 45 +++++++++++++++++++++++++++++-------------
 2 files changed, 33 insertions(+), 14 deletions(-)
include/linux/ioport.h +2 -0
index 5b6a7121c9f0..7bddddfc76d6 100644
@@ -297,6 +297,8 @@ static inline bool resource_overlaps(struct resource *r1, struct resource *r2)
 
 struct resource *devm_request_free_mem_region(struct device *dev,
 		struct resource *base, unsigned long size);
+struct resource *request_free_mem_region(struct resource *base,
+		unsigned long size, const char *name);
 
 #endif /* __ASSEMBLY__ */
 #endif	/* _LINUX_IOPORT_H */
kernel/resource.c +31 -14

64 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] resource: add a not device managed request_free_mem_region variant

Quoted 72 lines · Christoph Hellwig

Factor out the guts of devm_request_free_mem_region so that we can implement both a device managed and a manually release version as tiny wrappers around it.

---
 include/linux/ioport.h |  2 ++
 kernel/resource.c      | 45 +++++++++++++++++++++++++++++-------------
 2 files changed, 33 insertions(+), 14 deletions(-)
include/linux/ioport.h +2 -0
index 5b6a7121c9f0..7bddddfc76d6 100644
@@ -297,6 +297,8 @@ static inline bool resource_overlaps(struct resource *r1, struct resource *r2)
 
 struct resource *devm_request_free_mem_region(struct device *dev,
                struct resource *base, unsigned long size);
+struct resource *request_free_mem_region(struct resource *base,
+               unsigned long size, const char *name);
 
 #endif /* __ASSEMBLY__ */
 #endif /* _LINUX_IOPORT_H */
kernel/resource.c +11 -14
index 7ea4306503c5..74877e9d90ca 100644
@@ -1644,19 +1644,8 @@ void resource_list_free(struct list_head *head)
 EXPORT_SYMBOL(resource_list_free);
 
 #ifdef CONFIG_DEVICE_PRIVATE
-/**
- * devm_request_free_mem_region - find free region for device private memory
- *
- * @dev: device struct to bind the resource to
- * @size: size in bytes of the device memory to add
- * @base: resource tree to look in
- *
- * This function tries to find an empty range of physical address big enough to
- * contain the new resource, so that it can later be hotplugged as ZONE_DEVICE
- * memory, which in turn allocates struct pages.
- */
-struct resource *devm_request_free_mem_region(struct device *dev,
-               struct resource *base, unsigned long size)
+static struct resource *__request_free_mem_region(struct device *dev,
+               struct resource *base, unsigned long size, const char *name)
 {
        resource_size_t end, addr;
        struct resource *res;
@@ -1670,7 +1659,10 @@ struct resource *devm_request_free_mem_region(struct device *dev,
                                REGION_DISJOINT)
                        continue;
 
-               res = devm_request_mem_region(dev, addr, size, dev_name(dev));
+               if (dev)
+                       res = devm_request_mem_region(dev, addr, size, name);
+               else
+                       res = request_mem_region(addr, size, name);
                if (!res)
                        return ERR_PTR(-ENOMEM);
                res->desc = IORES_DESC_DEVICE_PRIVATE_MEMORY;
@@ -1679,7 +1671,32 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 
        return ERR_PTR(-ERANGE);
 }
+
+/**
+ * devm_request_free_mem_region - find free region for device private memory
+ *
+ * @dev: device struct to bind the resource to

kernel/resource.c struct resource *devm_request_free_mem_region(struct device *dev, · line 1688

Quoted 10 lines
kernel/resource.c +10 -0
+ * @size: size in bytes of the device memory to add
+ * @base: resource tree to look in
+ *
+ * This function tries to find an empty range of physical address big enough to
+ * contain the new resource, so that it can later be hotplugged as ZONE_DEVICE
+ * memory, which in turn allocates struct pages.
+ */
+struct resource *devm_request_free_mem_region(struct device *dev,
+               struct resource *base, unsigned long size)
+{

Previously we would loudly crash if someone passed NULL to devm_request_free_mem_region(), but now it will silently work and the result will leak. Perhaps this wants a:

if (!dev)

    return NULL;

...to head off those mistakes?

No major heartburn if you keep it as is, you can add:

#3 lore raw

Re: [PATCH 1/4] resource: add a not device managed request_free_mem_region variant

Quoted 4 lines · Dan Williams

Previously we would loudly crash if someone passed NULL to devm_request_free_mem_region(), but now it will silently work and the result will leak. Perhaps this wants a:

We'd still instantly crash due to the dev_name dereference, right?

#4 lore raw

Re: [PATCH 1/4] resource: add a not device managed request_free_mem_region variant

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Dan Williams

Previously we would loudly crash if someone passed NULL to devm_request_free_mem_region(), but now it will silently work and the result will leak. Perhaps this wants a:

We'd still instantly crash due to the dev_name dereference, right?

Whoops, yes.

#5 lore raw

PATCH 2/4 memremap: remove the dev field in struct dev_pagemap

In reply to Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

---
 include/linux/memremap.h | 1 -
 kernel/memremap.c        | 6 +-----
 mm/page_alloc.c          | 2 +-
 3 files changed, 2 insertions(+), 7 deletions(-)
include/linux/memremap.h +0 -1
index f8a5b2a19945..8f0013e18e14 100644
@@ -109,7 +109,6 @@ struct dev_pagemap {
 	struct percpu_ref *ref;
 	struct percpu_ref internal_ref;
 	struct completion done;
-	struct device *dev;
 	enum memory_type type;
 	unsigned int flags;
 	u64 pci_p2pdma_bus_offset;
kernel/memremap.c +1 -5
index 6ee03a816d67..600a14cbe663 100644
@@ -96,7 +96,6 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
 static void devm_memremap_pages_release(void *data)
 {
 	struct dev_pagemap *pgmap = data;
-	struct device *dev = pgmap->dev;
 	struct resource *res = &pgmap->res;
 	unsigned long pfn;
 	int nid;
@@ -123,8 +122,7 @@ static void devm_memremap_pages_release(void *data)
 
 	untrack_pfn(NULL, PHYS_PFN(res->start), resource_size(res));
 	pgmap_array_delete(res);
-	dev_WARN_ONCE(dev, pgmap->altmap.alloc,
-		      "%s: failed to free all reserved pages\n", __func__);
+	WARN_ONCE(pgmap->altmap.alloc, "failed to free all reserved pages\n");
 }
 
 static void dev_pagemap_percpu_release(struct percpu_ref *ref)
@@ -245,8 +243,6 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
 		goto err_array;
 	}
 
-	pgmap->dev = dev;
-
 	error = xa_err(xa_store_range(&pgmap_array, PHYS_PFN(res->start),
 				PHYS_PFN(res->end), pgmap, GFP_KERNEL));
 	if (error)
mm/page_alloc.c +1 -1
index 272c6de1bf4e..b39baa2b1faf 100644
@@ -5982,7 +5982,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 		}
 	}
 
-	pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev),
+	pr_info("%s initialised %lu pages in %ums\n", __func__,
 		size, jiffies_to_msecs(jiffies - start));
 }
 
Signature · 1 line
2.20.1
Reviewed-by on v1 2/4
#6 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

You can otherwise add:

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

---

(quoted diff) +0 -0

b/tools/testing/nvdimm/test/iomap.c
index cd040b5abffe..3f55f2f99112 100644

tools/testing/nvdimm/test/iomap.c +0 -0
@@ -132,7 +132,6 @@ void *__wrap_devm_memremap_pages(struct device

*dev, struct dev_pagemap *pgmap)

        if (!nfit_res)
                return devm_memremap_pages(dev, pgmap);
-       pgmap->dev = dev;
        if (!pgmap->ref) {
                if (pgmap->ops && (pgmap->ops->kill || pgmap->ops->cleanup))
                        return ERR_PTR(-EINVAL);
#7 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 15 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

Quoted 5 lines

You can otherwise add:

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

Can you get this merged? Do you want it to go with this series?

Jason

#8 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 17 lines · Jason Gunthorpe
Quoted 15 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

Quoted 7 lines
Quoted 5 lines

You can otherwise add:

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

Can you get this merged? Do you want it to go with this series?

Yeah, makes some sense to let you merge it so that you can get kbuild-robot reports about any follow-on memremap_pages() work that may trip up the build. Otherwise let me know and I'll get it queued with the other v5.4 libnvdimm pending bits.

#9 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 22 lines · Dan Williams
Quoted 21 lines · Jason Gunthorpe
Quoted 19 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

You can otherwise add:

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

Can you get this merged? Do you want it to go with this series?

Yeah, makes some sense to let you merge it so that you can get kbuild-robot reports about any follow-on memremap_pages() work that may trip up the build. Otherwise let me know and I'll get it queued with the other v5.4 libnvdimm pending bits.

Done, I used it already to test build the last series from CH..

Jason

#10 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 25 lines · Jason Gunthorpe
Quoted 24 lines · Dan Williams
Quoted 23 lines · Jason Gunthorpe
Quoted 20 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

You can otherwise add:

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Can you get this merged? Do you want it to go with this series?

Yeah, makes some sense to let you merge it so that you can get kbuild-robot reports about any follow-on memremap_pages() work that may trip up the build. Otherwise let me know and I'll get it queued with the other v5.4 libnvdimm pending bits.

Done, I used it already to test build the last series from CH..

It failed 0-day, I'm guessing some missing kconfig stuff

For now I dropped it, but, if you send a v2 I can forward it toward 0-day again!

Thanks,
Jason

#11 lore raw

Re: [PATCH 2/4] memremap: remove the dev field in struct dev_pagemap

Quoted 34 lines · Jason Gunthorpe
Quoted 32 lines · Jason Gunthorpe
Quoted 30 lines · Dan Williams
Quoted 17 lines · Jason Gunthorpe
Quoted 15 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

Needs the below as well.

/me goes to check if he ever merged the fix to make the unit test stuff get built by default with COMPILE_TEST [1]. Argh! Nope, didn't submit it for 5.3-rc1, sorry for the thrash.

Quoted 7 lines
Quoted 5 lines

You can otherwise add:

[1]: https://lore.kernel.org/lkml/156097224232.1086847.9463861924683372741.stgit@dwillia2-desk3.amr.corp.intel.com/

Can you get this merged? Do you want it to go with this series?

Yeah, makes some sense to let you merge it so that you can get kbuild-robot reports about any follow-on memremap_pages() work that may trip up the build. Otherwise let me know and I'll get it queued with the other v5.4 libnvdimm pending bits.

Quoted 6 lines
Quoted 1 line

Done, I used it already to test build the last series from CH..

It failed 0-day, I'm guessing some missing kconfig stuff

For now I dropped it, but, if you send a v2 I can forward it toward 0-day again!

The system works!

Sorry for that thrash, I'll track it down.

#12 lore raw

PATCH 3/4 memremap: don't use a separate devm action for devmap_managed_enable_get

In reply to Christoph Hellwig

Just clean up for early failures and then piggy back on devm_memremap_pages_release. This helps with a pending not device managed version of devm_memremap_pages.

---
 kernel/memremap.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
kernel/memremap.c +10 -5
index 600a14cbe663..09a087ca30ff 100644
@@ -21,13 +21,13 @@ DEFINE_STATIC_KEY_FALSE(devmap_managed_key);
 EXPORT_SYMBOL(devmap_managed_key);
 static atomic_t devmap_managed_enable;
 
-static void devmap_managed_enable_put(void *data)
+static void devmap_managed_enable_put(void)
 {
 	if (atomic_dec_and_test(&devmap_managed_enable))
 		static_branch_disable(&devmap_managed_key);
 }
 
-static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgmap)
+static int devmap_managed_enable_get(struct dev_pagemap *pgmap)
 {
 	if (!pgmap->ops || !pgmap->ops->page_free) {
 		WARN(1, "Missing page_free method\n");
@@ -36,13 +36,16 @@ static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgm
 
 	if (atomic_inc_return(&devmap_managed_enable) == 1)
 		static_branch_enable(&devmap_managed_key);
-	return devm_add_action_or_reset(dev, devmap_managed_enable_put, NULL);
+	return 0;
 }
 #else
-static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgmap)
+static int devmap_managed_enable_get(struct dev_pagemap *pgmap)
 {
 	return -EINVAL;
 }
+static void devmap_managed_enable_put(void)
+{
+}
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
 static void pgmap_array_delete(struct resource *res)
@@ -123,6 +126,7 @@ static void devm_memremap_pages_release(void *data)
 	untrack_pfn(NULL, PHYS_PFN(res->start), resource_size(res));
 	pgmap_array_delete(res);
 	WARN_ONCE(pgmap->altmap.alloc, "failed to free all reserved pages\n");
+	devmap_managed_enable_put();
 }
 
 static void dev_pagemap_percpu_release(struct percpu_ref *ref)
@@ -212,7 +216,7 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
 	}
 
 	if (need_devmap_managed) {
-		error = devmap_managed_enable_get(dev, pgmap);
+		error = devmap_managed_enable_get(pgmap);
 		if (error)
 			return ERR_PTR(error);
 	}
@@ -321,6 +325,7 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
  err_array:
 	dev_pagemap_kill(pgmap);
 	dev_pagemap_cleanup(pgmap);
+	devmap_managed_enable_put();
 	return ERR_PTR(error);
 }
 EXPORT_SYMBOL_GPL(devm_memremap_pages);
Signature · 1 line
2.20.1
Reviewed-by on v1 3/4
#13 lore raw

Re: [PATCH 3/4] memremap: don't use a separate devm action for devmap_managed_enable_get

Quoted 7 lines · Christoph Hellwig

Just clean up for early failures and then piggy back on devm_memremap_pages_release. This helps with a pending not device managed version of devm_memremap_pages.

Looks good,

#14 lore raw

PATCH 4/4 memremap: provide a not device managed memremap_pages

In reply to Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it.

---
 include/linux/memremap.h |  2 +
 kernel/memremap.c        | 84 +++++++++++++++++++++++++---------------
 2 files changed, 54 insertions(+), 32 deletions(-)
include/linux/memremap.h +2 -0
index 8f0013e18e14..fb2a0bd826b9 100644
@@ -123,6 +123,8 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap)
 }
 
 #ifdef CONFIG_ZONE_DEVICE
+void *memremap_pages(struct dev_pagemap *pgmap, int nid);
+void memunmap_pages(struct dev_pagemap *pgmap);
 void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap);
 void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap);
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
kernel/memremap.c +52 -32

140 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v1 4/4
#15 lore raw

Re: [PATCH 4/4] memremap: provide a not device managed memremap_pages

Quoted 7 lines · Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it.

Looks good,

Tested-by
#16 lore raw
In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig

Hi Dan and Jason,

Bharata has been working on secure page management for kvmppc guests, and one I thing I noticed is that he had to fake up a struct device just so that it could be passed to the devm_memremap_pages instrastructure for device private memory.

This series adds non-device managed versions of the devm_request_free_mem_region and devm_memremap_pages functions for his use case.

Tested kvmppc ultravisor patchset with migrate_vma changes and this patchset. (Had to manually patch mm/memremap.c instead of kernel/memremap.c though)

For the series,

#17 lore raw
Quoted 8 lines · Bharata B Rao
Quoted 7 lines · Christoph Hellwig

Hi Dan and Jason,

Bharata has been working on secure page management for kvmppc guests, and one I thing I noticed is that he had to fake up a struct device just so that it could be passed to the devm_memremap_pages instrastructure for device private memory.

Quoted 7 lines
Quoted 3 lines

This series adds non-device managed versions of the devm_request_free_mem_region and devm_memremap_pages functions for his use case.

Tested kvmppc ultravisor patchset with migrate_vma changes and this patchset. (Had to manually patch mm/memremap.c instead of kernel/memremap.c though)

Oh. I rebased to the hmm tree, and that didn't have the rename yet. And I didn't even notice that as git handled it transparently.

#18 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig

Hi Dan and Jason,

Bharata has been working on secure page management for kvmppc guests, and one I thing I noticed is that he had to fake up a struct device just so that it could be passed to the devm_memremap_pages instrastructure for device private memory.

This series adds non-device managed versions of the devm_request_free_mem_region and devm_memremap_pages functions for his use case.

Quoted 6 lines

Changes since v2:
- improved changelogs that the the v2 changes into account

Changes since v1:
- don't overload devm_request_free_mem_region
- export the memremap_pages and munmap_pages as kvmppc can be a module

Looks good, I fixed up the patch with Dan's note and reviewed them as well.

Applied to hmm.git as requested

Thanks,
Jason