Skip to contentSkip to conversation

RFC: dev_pagemap reference counting

PATCH In review Christoph Hellwig 2 participants 6 messages Dec 5, 2017 – Dec 6, 2017

Subsystems (4)
lore
Share & tools
lore
#1 lore raw

PATCH 1/2 mm: move get_dev_pagemap out of line

This is a pretty big function, which should be out of line in general, and a no-op stub if CONFIG_ZONE_DEVICЕ is not set.

---
 include/linux/memremap.h | 42 +++++-------------------------------------
 kernel/memremap.c        | 36 ++++++++++++++++++++++++++++++++++--
 2 files changed, 39 insertions(+), 39 deletions(-)
include/linux/memremap.h +5 -37
index 10d23c367048..f24e0c71d6a6 100644
@@ -136,8 +136,8 @@ struct dev_pagemap {
 #ifdef CONFIG_ZONE_DEVICE
 void *devm_memremap_pages(struct device *dev, struct resource *res,
 		struct percpu_ref *ref, struct vmem_altmap *altmap);
-struct dev_pagemap *find_dev_pagemap(resource_size_t phys);
-
+struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+		struct dev_pagemap *pgmap);
 static inline bool is_zone_device_page(const struct page *page);
 #else
 static inline void *devm_memremap_pages(struct device *dev,
@@ -153,11 +153,12 @@ static inline void *devm_memremap_pages(struct device *dev,
 	return ERR_PTR(-ENXIO);
 }
 
-static inline struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
+static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+		struct dev_pagemap *pgmap)
 {
 	return NULL;
 }
-#endif
+#endif /* CONFIG_ZONE_DEVICE */
 
 #if defined(CONFIG_DEVICE_PRIVATE) || defined(CONFIG_DEVICE_PUBLIC)
 static inline bool is_device_private_page(const struct page *page)
@@ -173,39 +174,6 @@ static inline bool is_device_public_page(const struct page *page)
 }
 #endif /* CONFIG_DEVICE_PRIVATE || CONFIG_DEVICE_PUBLIC */
 
-/**
- * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn
- * @pfn: page frame number to lookup page_map
- * @pgmap: optional known pgmap that already has a reference
- *
- * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
- * same mapping.
- */
-static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
-		struct dev_pagemap *pgmap)
-{
-	const struct resource *res = pgmap ? pgmap->res : NULL;
-	resource_size_t phys = PFN_PHYS(pfn);
-
-	/*
-	 * In the cached case we're already holding a live reference so
-	 * we can simply do a blind increment
-	 */
-	if (res && phys >= res->start && phys <= res->end) {
-		percpu_ref_get(pgmap->ref);
-		return pgmap;
-	}
-
-	/* fall back to slow path lookup */
-	rcu_read_lock();
-	pgmap = find_dev_pagemap(phys);
-	if (pgmap && !percpu_ref_tryget_live(pgmap->ref))
-		pgmap = NULL;
-	rcu_read_unlock();
-
-	return pgmap;
-}
-
 static inline void put_dev_pagemap(struct dev_pagemap *pgmap)
 {
 	if (pgmap)
kernel/memremap.c +34 -2
index 403ab9cdb949..f0b54eca85b0 100644
@@ -314,7 +314,7 @@ static void devm_memremap_pages_release(struct device *dev, void *data)
 }
 
 /* assumes rcu_read_lock() held at entry */
-struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
+static struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
 {
 	struct page_map *page_map;
 
@@ -500,8 +500,40 @@ struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
 
 	return pgmap ? pgmap->altmap : NULL;
 }
-#endif /* CONFIG_ZONE_DEVICE */
 
+/**
+ * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn
+ * @pfn: page frame number to lookup page_map
+ * @pgmap: optional known pgmap that already has a reference
+ *
+ * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
+ * same mapping.
+ */
+struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
+		struct dev_pagemap *pgmap)
+{
+	const struct resource *res = pgmap ? pgmap->res : NULL;
+	resource_size_t phys = PFN_PHYS(pfn);
+
+	/*
+	 * In the cached case we're already holding a live reference so
+	 * we can simply do a blind increment
+	 */
+	if (res && phys >= res->start && phys <= res->end) {
+		percpu_ref_get(pgmap->ref);
+		return pgmap;
+	}
+
+	/* fall back to slow path lookup */
+	rcu_read_lock();
+	pgmap = find_dev_pagemap(phys);
+	if (pgmap && !percpu_ref_tryget_live(pgmap->ref))
+		pgmap = NULL;
+	rcu_read_unlock();
+
+	return pgmap;
+}
+#endif /* CONFIG_ZONE_DEVICE */
 
 #if IS_ENABLED(CONFIG_DEVICE_PRIVATE) ||  IS_ENABLED(CONFIG_DEVICE_PUBLIC)
 void put_zone_device_private_or_public_page(struct page *page)
Signature · 6 lines
2.14.2

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#2 lore raw

PATCH 2/2 mm: fix dev_pagemap reference counting around get_dev_pagemap

In reply to Christoph Hellwig

Both callers of get_dev_pagemap that pass in a pgmap don't actually hold a reference to the pgmap they pass in, contrary to the comment in the function.

Change the calling convention so that get_dev_pagemap always consumes the previous reference instead of doing this using an explicit earlier call to put_dev_pagemap in the callers.

The callers will still need to put the final reference after finishing the loop over the pages.

---
 kernel/memremap.c | 17 +++++++++--------
 mm/gup.c          |  7 +++++--
 2 files changed, 14 insertions(+), 10 deletions(-)
kernel/memremap.c +9 -8
index f0b54eca85b0..502fa107a585 100644
@@ -506,22 +506,23 @@ struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
  * @pfn: page frame number to lookup page_map
  * @pgmap: optional known pgmap that already has a reference
  *
- * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
- * same mapping.
+ * If @pgmap is non-NULL and covers @pfn it will be returned as-is.  If @pgmap
+ * is non-NULL but does not cover @pfn the reference to it while be released.
  */
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
 		struct dev_pagemap *pgmap)
 {
-	const struct resource *res = pgmap ? pgmap->res : NULL;
 	resource_size_t phys = PFN_PHYS(pfn);
 
 	/*
-	 * In the cached case we're already holding a live reference so
-	 * we can simply do a blind increment
+	 * In the cached case we're already holding a live reference.
 	 */
-	if (res && phys >= res->start && phys <= res->end) {
-		percpu_ref_get(pgmap->ref);
-		return pgmap;
+	if (pgmap) {
+		const struct resource *res = pgmap ? pgmap->res : NULL;
+
+		if (res && phys >= res->start && phys <= res->end)
+			return pgmap;
+		put_dev_pagemap(pgmap);
 	}
 
 	/* fall back to slow path lookup */
mm/gup.c +5 -2
index d3fb60e5bfac..9d142eb9e2e9 100644
@@ -1410,7 +1410,6 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
 
 		VM_BUG_ON_PAGE(compound_head(page) != head, page);
 
-		put_dev_pagemap(pgmap);
 		SetPageReferenced(page);
 		pages[*nr] = page;
 		(*nr)++;
@@ -1420,6 +1419,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
 	ret = 1;
 
 pte_unmap:
+	if (pgmap)
+		put_dev_pagemap(pgmap);
 	pte_unmap(ptem);
 	return ret;
 }
@@ -1459,10 +1460,12 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr,
 		SetPageReferenced(page);
 		pages[*nr] = page;
 		get_page(page);
-		put_dev_pagemap(pgmap);
 		(*nr)++;
 		pfn++;
 	} while (addr += PAGE_SIZE, addr != end);
+
+	if (pgmap)
+		put_dev_pagemap(pgmap);
 	return 1;
 }
 
Signature · 6 lines
2.14.2

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#3 lore raw

Re: [PATCH 2/2] mm: fix dev_pagemap reference counting around get_dev_pagemap

Quoted 9 lines · Christoph Hellwig

Both callers of get_dev_pagemap that pass in a pgmap don't actually hold a reference to the pgmap they pass in, contrary to the comment in the function.

Change the calling convention so that get_dev_pagemap always consumes the previous reference instead of doing this using an explicit earlier call to put_dev_pagemap in the callers.

The callers will still need to put the final reference after finishing the loop over the pages.

I don't think we need this change, but perhaps the reasoning should be added to the code as a comment... details below.

Quoted 65 lines
---
 kernel/memremap.c | 17 +++++++++--------
 mm/gup.c          |  7 +++++--
 2 files changed, 14 insertions(+), 10 deletions(-)
kernel/memremap.c +9 -8
index f0b54eca85b0..502fa107a585 100644
@@ -506,22 +506,23 @@ struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
  * @pfn: page frame number to lookup page_map
  * @pgmap: optional known pgmap that already has a reference
  *
- * @pgmap allows the overhead of a lookup to be bypassed when @pfn lands in the
- * same mapping.
+ * If @pgmap is non-NULL and covers @pfn it will be returned as-is.  If @pgmap
+ * is non-NULL but does not cover @pfn the reference to it while be released.
  */
 struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
                struct dev_pagemap *pgmap)
 {
-       const struct resource *res = pgmap ? pgmap->res : NULL;
        resource_size_t phys = PFN_PHYS(pfn);
 
        /*
-        * In the cached case we're already holding a live reference so
-        * we can simply do a blind increment
+        * In the cached case we're already holding a live reference.
         */
-       if (res && phys >= res->start && phys <= res->end) {
-               percpu_ref_get(pgmap->ref);
-               return pgmap;
+       if (pgmap) {
+               const struct resource *res = pgmap ? pgmap->res : NULL;
+
+               if (res && phys >= res->start && phys <= res->end)
+                       return pgmap;
+               put_dev_pagemap(pgmap);
        }
 
        /* fall back to slow path lookup */
mm/gup.c +2 -1
index d3fb60e5bfac..9d142eb9e2e9 100644
@@ -1410,7 +1410,6 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
 
                VM_BUG_ON_PAGE(compound_head(page) != head, page);
 
-               put_dev_pagemap(pgmap);
                SetPageReferenced(page);
                pages[*nr] = page;
                (*nr)++;
@@ -1420,6 +1419,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
        ret = 1;
 
 pte_unmap:
+       if (pgmap)
+               put_dev_pagemap(pgmap);
        pte_unmap(ptem);
        return ret;
 }

mm/gup.c static int __gup_device_huge(unsigned long pfn, unsigned long addr, · line 1462

Quoted 4 lines
mm/gup.c +0 -1
@@ -1459,10 +1460,12 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr,
                SetPageReferenced(page);
                pages[*nr] = page;
                get_page(page);
-               put_dev_pagemap(pgmap);

It's safe to do the put_dev_pagemap() here because the pgmap cannot be released until the corresponding put_page() for that get_page() we just did occurs. So we're only holding the pgmap reference long enough to take individual page references.

We used to take and put individual pgmap references inside get_page() / put_page(), but that got simplified in this commit to just take and put page reference at devm_memremap_pages() setup / teardown time:

71389703839e mm, zone_device: Replace {get, put}_zone_device_page() with a single reference to fix pmem crash

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

Re: [PATCH 2/2] mm: fix dev_pagemap reference counting around get_dev_pagemap

Quoted 2 lines · Dan Williams

I don't think we need this change, but perhaps the reasoning should be added to the code as a comment... details below.

Hmm, looks like we are ok at least. But even if it's not a correctness issue there is no good point in decrementing and incrementing the reference count every time.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#5 lore raw

Re: [PATCH 2/2] mm: fix dev_pagemap reference counting around get_dev_pagemap

Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Dan Williams

I don't think we need this change, but perhaps the reasoning should be added to the code as a comment... details below.

Hmm, looks like we are ok at least. But even if it's not a correctness issue there is no good point in decrementing and incrementing the reference count every time.

True, we can take it once and drop it at the end when all the related page references have been taken.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm