Skip to contentSkip to conversation

add a not device managed memremap_pages

PATCH In review Christoph Hellwig 3 participants 16 messages Aug 11, 2019 – Aug 14, 2019

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

Hi Dan and Jason,

Bharata has been working on secure page management for kvmppc guests, and one I thing I noticed is that he had to fake up a struct device just so that it could be passed to the devm_memremap_pages instrastructure for device private memory.

This series adds non-device managed versions of the devm_request_free_mem_region and devm_memremap_pages functions for his use case.

#1 lore raw

PATCH 1/5 resource: pass a name argument to devm_request_free_mem_region

Add an explicit resource name argument to devm_request_free_mem_region. Besides allowing drivers to request multiple regions per device with different names, this also prepares for a not device managed version of the function.

---
 drivers/gpu/drm/nouveau/nouveau_dmem.c | 3 ++-
 include/linux/ioport.h                 | 2 +-
 kernel/resource.c                      | 5 +++--
 3 files changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/nouveau/nouveau_dmem.c +2 -1
index 1333220787a1..aedf18a44789 100644
@@ -605,7 +605,8 @@ nouveau_dmem_init(struct nouveau_drm *drm)
 	 * and latter if we want to do thing like over commit then we
 	 * could revisit this.
 	 */
-	res = devm_request_free_mem_region(device, &iomem_resource, size);
+	res = devm_request_free_mem_region(device, &iomem_resource, size,
+			dev_name(device));
 	if (IS_ERR(res))
 		goto out_free;
 	drm->dmem->pagemap.type = MEMORY_DEVICE_PRIVATE;
include/linux/ioport.h +1 -1
index 5b6a7121c9f0..0dcc48cafa80 100644
@@ -296,7 +296,7 @@ static inline bool resource_overlaps(struct resource *r1, struct resource *r2)
 }
 
 struct resource *devm_request_free_mem_region(struct device *dev,
-		struct resource *base, unsigned long size);
+		struct resource *base, unsigned long size, const char *name);
 
 #endif /* __ASSEMBLY__ */
 #endif	/* _LINUX_IOPORT_H */
kernel/resource.c +3 -2
index 7ea4306503c5..0ddc558586a7 100644
@@ -1650,13 +1650,14 @@ EXPORT_SYMBOL(resource_list_free);
  * @dev: device struct to bind the resource to
  * @size: size in bytes of the device memory to add
  * @base: resource tree to look in
+ * @name: identifying name for the new resource
  *
  * This function tries to find an empty range of physical address big enough to
  * contain the new resource, so that it can later be hotplugged as ZONE_DEVICE
  * memory, which in turn allocates struct pages.
  */
 struct resource *devm_request_free_mem_region(struct device *dev,
-		struct resource *base, unsigned long size)
+		struct resource *base, unsigned long size, const char *name)
 {
 	resource_size_t end, addr;
 	struct resource *res;
@@ -1670,7 +1671,7 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 				REGION_DISJOINT)
 			continue;
 
-		res = devm_request_mem_region(dev, addr, size, dev_name(dev));
+		res = devm_request_mem_region(dev, addr, size, name);
 		if (!res)
 			return ERR_PTR(-ENOMEM);
 		res->desc = IORES_DESC_DEVICE_PRIVATE_MEMORY;
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/5 resource: add a not device managed request_free_mem_region variant

In reply to Christoph Hellwig

Just add a simple macro that passes a NULL dev argument to dev_request_free_mem_region, and call request_mem_region in the function for that particular case.

---
 include/linux/ioport.h | 2 ++
 kernel/resource.c      | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/ioport.h +2 -0
index 0dcc48cafa80..528ae6cbb1b4 100644
@@ -297,6 +297,8 @@ static inline bool resource_overlaps(struct resource *r1, struct resource *r2)
 
 struct resource *devm_request_free_mem_region(struct device *dev,
 		struct resource *base, unsigned long size, const char *name);
+#define request_free_mem_region(base, size, name) \
+	devm_request_free_mem_region(NULL, base, size, name)
 
 #endif /* __ASSEMBLY__ */
 #endif	/* _LINUX_IOPORT_H */
kernel/resource.c +4 -1
index 0ddc558586a7..3a826b3cc883 100644
@@ -1671,7 +1671,10 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 				REGION_DISJOINT)
 			continue;
 
-		res = devm_request_mem_region(dev, addr, size, name);
+		if (dev)
+			res = devm_request_mem_region(dev, addr, size, name);
+		else
+			res = request_mem_region(addr, size, name);
 		if (!res)
 			return ERR_PTR(-ENOMEM);
 		res->desc = IORES_DESC_DEVICE_PRIVATE_MEMORY;
Signature · 1 line
2.20.1
#3 lore raw

Re: [PATCH 2/5] resource: add a not device managed request_free_mem_region variant

Quoted 25 lines · Christoph Hellwig

Just add a simple macro that passes a NULL dev argument to dev_request_free_mem_region, and call request_mem_region in the function for that particular case.

 include/linux/ioport.h | 2 ++
 kernel/resource.c      | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/ioport.h +3 -0
index 0dcc48cafa80..528ae6cbb1b4 100644
+++ b/include/linux/ioport.h
@@ -297,6 +297,8 @@ static inline bool resource_overlaps(struct resource *r1, struct resource *r2)
 
 struct resource *devm_request_free_mem_region(struct device *dev,
 		struct resource *base, unsigned long size, const char *name);
+#define request_free_mem_region(base, size, name) \
+	devm_request_free_mem_region(NULL, base, size, name)
 
 #endif /* __ASSEMBLY__ */
 #endif	/* _LINUX_IOPORT_H */
kernel/resource.c +1 -0
index 0ddc558586a7..3a826b3cc883 100644
+++ b/kernel/resource.c

kernel/resource.c struct resource *devm_request_free_mem_region(struct device *dev, · line 1677

Quoted 8 lines
kernel/resource.c +4 -1
@@ -1671,7 +1671,10 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 				REGION_DISJOINT)
 			continue;
 
-		res = devm_request_mem_region(dev, addr, size, name);
+		if (dev)
+			res = devm_request_mem_region(dev, addr, size, name);
+		else
+			res = request_mem_region(addr, size, name);

It is a bit jarring to have something called devm_* that doesn't actually do the devm_ part on some paths.

Maybe this function should be called __request_free_mem_region() with another name wrapper macro?

Jason

#4 lore raw

Re: [PATCH 2/5] resource: add a not device managed request_free_mem_region variant

Quoted 5 lines · Jason Gunthorpe

It is a bit jarring to have something called devm_* that doesn't actually do the devm_ part on some paths.

Maybe this function should be called __request_free_mem_region() with another name wrapper macro?

Seems like a little more churn than required, but I could do it.

#5 lore raw

PATCH 3/5 memremap: remove the dev field in struct dev_pagemap

In reply to Christoph Hellwig

The dev field in struct dev_pagemap is only used to print dev_name in two places, which are at best nice to have. Just remove the field and thus the name in those two messages.

---
 include/linux/memremap.h | 1 -
 mm/memremap.c            | 6 +-----
 mm/page_alloc.c          | 2 +-
 3 files changed, 2 insertions(+), 7 deletions(-)
include/linux/memremap.h +0 -1
index f8a5b2a19945..8f0013e18e14 100644
@@ -109,7 +109,6 @@ struct dev_pagemap {
 	struct percpu_ref *ref;
 	struct percpu_ref internal_ref;
 	struct completion done;
-	struct device *dev;
 	enum memory_type type;
 	unsigned int flags;
 	u64 pci_p2pdma_bus_offset;
mm/memremap.c +1 -5
index 6ee03a816d67..600a14cbe663 100644
@@ -96,7 +96,6 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
 static void devm_memremap_pages_release(void *data)
 {
 	struct dev_pagemap *pgmap = data;
-	struct device *dev = pgmap->dev;
 	struct resource *res = &pgmap->res;
 	unsigned long pfn;
 	int nid;
@@ -123,8 +122,7 @@ static void devm_memremap_pages_release(void *data)
 
 	untrack_pfn(NULL, PHYS_PFN(res->start), resource_size(res));
 	pgmap_array_delete(res);
-	dev_WARN_ONCE(dev, pgmap->altmap.alloc,
-		      "%s: failed to free all reserved pages\n", __func__);
+	WARN_ONCE(pgmap->altmap.alloc, "failed to free all reserved pages\n");
 }
 
 static void dev_pagemap_percpu_release(struct percpu_ref *ref)
@@ -245,8 +243,6 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
 		goto err_array;
 	}
 
-	pgmap->dev = dev;
-
 	error = xa_err(xa_store_range(&pgmap_array, PHYS_PFN(res->start),
 				PHYS_PFN(res->end), pgmap, GFP_KERNEL));
 	if (error)
mm/page_alloc.c +1 -1
index 272c6de1bf4e..b39baa2b1faf 100644
@@ -5982,7 +5982,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 		}
 	}
 
-	pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev),
+	pr_info("%s initialised %lu pages in %ums\n", __func__,
 		size, jiffies_to_msecs(jiffies - start));
 }
 
Signature · 1 line
2.20.1
#6 lore raw

PATCH 4/5 memremap: don't use a separate devm action for devmap_managed_enable_get

In reply to Christoph Hellwig

Just clean up for early failures and then piggy back on devm_memremap_pages_release. This helps with a pending not device managed version of devm_memremap_pages.

---
 mm/memremap.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
mm/memremap.c +10 -5
index 600a14cbe663..09a087ca30ff 100644
@@ -21,13 +21,13 @@ DEFINE_STATIC_KEY_FALSE(devmap_managed_key);
 EXPORT_SYMBOL(devmap_managed_key);
 static atomic_t devmap_managed_enable;
 
-static void devmap_managed_enable_put(void *data)
+static void devmap_managed_enable_put(void)
 {
 	if (atomic_dec_and_test(&devmap_managed_enable))
 		static_branch_disable(&devmap_managed_key);
 }
 
-static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgmap)
+static int devmap_managed_enable_get(struct dev_pagemap *pgmap)
 {
 	if (!pgmap->ops || !pgmap->ops->page_free) {
 		WARN(1, "Missing page_free method\n");
@@ -36,13 +36,16 @@ static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgm
 
 	if (atomic_inc_return(&devmap_managed_enable) == 1)
 		static_branch_enable(&devmap_managed_key);
-	return devm_add_action_or_reset(dev, devmap_managed_enable_put, NULL);
+	return 0;
 }
 #else
-static int devmap_managed_enable_get(struct device *dev, struct dev_pagemap *pgmap)
+static int devmap_managed_enable_get(struct dev_pagemap *pgmap)
 {
 	return -EINVAL;
 }
+static void devmap_managed_enable_put(void)
+{
+}
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
 static void pgmap_array_delete(struct resource *res)
@@ -123,6 +126,7 @@ static void devm_memremap_pages_release(void *data)
 	untrack_pfn(NULL, PHYS_PFN(res->start), resource_size(res));
 	pgmap_array_delete(res);
 	WARN_ONCE(pgmap->altmap.alloc, "failed to free all reserved pages\n");
+	devmap_managed_enable_put();
 }
 
 static void dev_pagemap_percpu_release(struct percpu_ref *ref)
@@ -212,7 +216,7 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
 	}
 
 	if (need_devmap_managed) {
-		error = devmap_managed_enable_get(dev, pgmap);
+		error = devmap_managed_enable_get(pgmap);
 		if (error)
 			return ERR_PTR(error);
 	}
@@ -321,6 +325,7 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)
  err_array:
 	dev_pagemap_kill(pgmap);
 	dev_pagemap_cleanup(pgmap);
+	devmap_managed_enable_put();
 	return ERR_PTR(error);
 }
 EXPORT_SYMBOL_GPL(devm_memremap_pages);
Signature · 1 line
2.20.1
#7 lore raw

PATCH 5/5 memremap: provide a not device managed memremap_pages

In reply to Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it. Note that this function is not exported, and doesn't have a cleanup routine associated with it to discourage use from more driver like users.

---
 include/linux/memremap.h |  1 +
 mm/memremap.c            | 74 ++++++++++++++++++++++++----------------
 2 files changed, 45 insertions(+), 30 deletions(-)
include/linux/memremap.h +1 -0
index 8f0013e18e14..eac23e88a94a 100644
@@ -123,6 +123,7 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap)
 }
 
 #ifdef CONFIG_ZONE_DEVICE
+void *memremap_pages(struct dev_pagemap *pgmap, int nid);
 void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap);
 void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap);
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
mm/memremap.c +44 -30

117 lines not on this page · raw message

Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 40 lines · Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it. Note that this function is not exported, and doesn't have a cleanup routine associated with it to discourage use from more driver like users.

 include/linux/memremap.h |  1 +
 mm/memremap.c            | 74 ++++++++++++++++++++++++----------------
 2 files changed, 45 insertions(+), 30 deletions(-)
include/linux/memremap.h +2 -0
index 8f0013e18e14..eac23e88a94a 100644
+++ b/include/linux/memremap.h
@@ -123,6 +123,7 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap)
 }
 
 #ifdef CONFIG_ZONE_DEVICE
+void *memremap_pages(struct dev_pagemap *pgmap, int nid);
 void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap);
 void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap);
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
mm/memremap.c +1 -11
index 09a087ca30ff..7b7575330db4 100644
+++ b/mm/memremap.c
@@ -137,27 +137,12 @@ static void dev_pagemap_percpu_release(struct percpu_ref *ref)
 	complete(&pgmap->done);
 }
 
-/**
- * devm_memremap_pages - remap and provide memmap backing for the given resource
- * @dev: hosting device for @res
- * @pgmap: pointer to a struct dev_pagemap
- *
- * Notes:
- * 1/ At a minimum the res and type members of @pgmap must be initialized
- *    by the caller before passing it to this function
- *
- * 2/ The altmap field may optionally be initialized, in which case
- *    PGMAP_ALTMAP_VALID must be set in pgmap->flags.

mm/memremap.c static void dev_pagemap_percpu_release(struct percpu_ref *ref) · line 141

Quoted 10 lines
mm/memremap.c +2 -8
- *
- * 3/ The ref field may optionally be provided, in which pgmap->ref must be
- *    'live' on entry and will be killed and reaped at
- *    devm_memremap_pages_release() time, or if this routine fails.
- *
- * 4/ res is expected to be a host memory range that could feasibly be
- *    treated as a "System RAM" range, i.e. not a device mmio range, but
- *    this is not enforced.
+/*
+ * This version is not intended for system resources only, and there is no

Was 'is not' what was intended here? I'm having a hard time reading this.

Jason

#9 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 4 lines · Jason Gunthorpe
Quoted 1 line

+ * This version is not intended for system resources only, and there is no

Was 'is not' what was intended here? I'm having a hard time reading this.

s/not//g

#10 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it. Note that this function is not exported, and doesn't have a cleanup routine associated with it to discourage use from more driver like users.

The kvmppc secure pages management code will be part of kvm-hv which can be built as module too. So it would require memremap_pages() to be exported.

Additionally, non-dev version of the cleanup routine devm_memremap_pages_release() or equivalent would also be requried. With device being present, put_device() used to take care of this cleanup.

Regards,
Bharata.

#11 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 7 lines · Bharata B Rao
Quoted 5 lines · Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it. Note that this function is not exported, and doesn't have a cleanup routine associated with it to discourage use from more driver like users.

Quoted 8 lines

The kvmppc secure pages management code will be part of kvm-hv which can be built as module too. So it would require memremap_pages() to be exported.

Additionally, non-dev version of the cleanup routine devm_memremap_pages_release() or equivalent would also be requried. With device being present, put_device() used to take care of this cleanup.

Oh well. We can add them fairly easily if we really need to, but I tried to avoid that. Can you try to see if this works non-modular for you for now until we hear more feedback from Dan?

#12 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Bharata B Rao
Quoted 5 lines · Christoph Hellwig

The kvmppc ultravisor code wants a device private memory pool that is system wide and not attached to a device. Instead of faking up one provide a low-level memremap_pages for it. Note that this function is not exported, and doesn't have a cleanup routine associated with it to discourage use from more driver like users.

The kvmppc secure pages management code will be part of kvm-hv which can be built as module too. So it would require memremap_pages() to be exported.

Quoted 8 lines
Quoted 4 lines

Additionally, non-dev version of the cleanup routine devm_memremap_pages_release() or equivalent would also be requried. With device being present, put_device() used to take care of this cleanup.

Oh well. We can add them fairly easily if we really need to, but I tried to avoid that. Can you try to see if this works non-modular for you for now until we hear more feedback from Dan?

Yes, this patchset works non-modular and with kvm-hv as module, it works with devm_memremap_pages_release() and release_mem_region() in the cleanup path. The cleanup path will be required in the non-modular case too for proper recovery from failures.

Regards,
Bharata.

#13 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 4 lines · Bharata B Rao

Yes, this patchset works non-modular and with kvm-hv as module, it works with devm_memremap_pages_release() and release_mem_region() in the cleanup path. The cleanup path will be required in the non-modular case too for proper recovery from failures.

Can you check if the version here:

git://git.infradead.org/users/hch/misc.git pgmap-remove-dev

Gitweb:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-remove-dev

works for you fully before I resend?

Quoted 3 lines

Regards,
Bharata.

---end quoted text---

#14 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Bharata B Rao

Yes, this patchset works non-modular and with kvm-hv as module, it works with devm_memremap_pages_release() and release_mem_region() in the cleanup path. The cleanup path will be required in the non-modular case too for proper recovery from failures.

Can you check if the version here:

Quoted 7 lines

git://git.infradead.org/users/hch/misc.git pgmap-remove-dev

Gitweb:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-remove-dev

works for you fully before I resend?

Yes, this works for us. This and migrate-vma-cleanup series helps to really simplify the kvmppc secure pages management code. Thanks.

Regards,
Bharata.

#15 lore raw

Re: [PATCH 5/5] memremap: provide a not device managed memremap_pages

Quoted 6 lines · Bharata B Rao
Quoted 3 lines

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-remove-dev

works for you fully before I resend?

Yes, this works for us. This and migrate-vma-cleanup series helps to really simplify the kvmppc secure pages management code. Thanks.

Thanks. I'm going to resend it once we've made a bit of progress on the migrate_vma series that I resent this morning. There are a few more lose ends in this area with implications for the driver API, so I might have a few more patches for you to test in a bit.