Skip to contentSkip to conversation

xen-gnttab: do not add m2p override entries for blkback mappings

PATCH In review Anthony Liguori 5 participants 9 messages Nov 13, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen-gnttab: do not add m2p override entries for blkback mappings

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

---
A backported version of this has been heavily tested but the testing against the latest Linux tree is light so far.

---
 drivers/block/xen-blkback/blkback.c |   12 ++++-----
 drivers/xen/gntdev.c                |    4 +--
 drivers/xen/grant-table.c           |   49 ++++++++++++++++++++++++++++-------
 include/xen/grant_table.h           |   14 ++++++++--
 4 files changed, 59 insertions(+), 20 deletions(-)
drivers/block/xen-blkback/blkback.c +6 -6
index bf4b9d2..53ea90e 100644
@@ -286,7 +286,7 @@ static void free_persistent_gnts(struct xen_blkif *blkif, struct rb_root *root,
 		if (++segs_to_unmap == BLKIF_MAX_SEGMENTS_PER_REQUEST ||
 			!rb_next(&persistent_gnt->node)) {
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
-				segs_to_unmap);
+						segs_to_unmap, 0);
 			BUG_ON(ret);
 			put_free_pages(blkif, pages, segs_to_unmap);
 			segs_to_unmap = 0;
@@ -322,7 +322,7 @@ static void unmap_purged_grants(struct work_struct *work)
 
 		if (++segs_to_unmap == BLKIF_MAX_SEGMENTS_PER_REQUEST) {
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
-				segs_to_unmap);
+						segs_to_unmap, 0);
 			BUG_ON(ret);
 			put_free_pages(blkif, pages, segs_to_unmap);
 			segs_to_unmap = 0;
@@ -330,7 +330,7 @@ static void unmap_purged_grants(struct work_struct *work)
 		kfree(persistent_gnt);
 	}
 	if (segs_to_unmap > 0) {
-		ret = gnttab_unmap_refs(unmap, NULL, pages, segs_to_unmap);
+		ret = gnttab_unmap_refs(unmap, NULL, pages, segs_to_unmap, 0);
 		BUG_ON(ret);
 		put_free_pages(blkif, pages, segs_to_unmap);
 	}
@@ -671,14 +671,14 @@ static void xen_blkbk_unmap(struct xen_blkif *blkif,
 		pages[i]->handle = BLKBACK_INVALID_HANDLE;
 		if (++invcount == BLKIF_MAX_SEGMENTS_PER_REQUEST) {
 			ret = gnttab_unmap_refs(unmap, NULL, unmap_pages,
-			                        invcount);
+			                        invcount, 0);
 			BUG_ON(ret);
 			put_free_pages(blkif, unmap_pages, invcount);
 			invcount = 0;
 		}
 	}
 	if (invcount) {
-		ret = gnttab_unmap_refs(unmap, NULL, unmap_pages, invcount);
+		ret = gnttab_unmap_refs(unmap, NULL, unmap_pages, invcount, 0);
 		BUG_ON(ret);
 		put_free_pages(blkif, unmap_pages, invcount);
 	}
@@ -740,7 +740,7 @@ again:
 	}
 
 	if (segs_to_map) {
-		ret = gnttab_map_refs(map, NULL, pages_to_gnt, segs_to_map);
+		ret = gnttab_map_refs(map, NULL, pages_to_gnt, segs_to_map, 0);
 		BUG_ON(ret);
 	}
 
drivers/xen/gntdev.c +2 -2
index e41c79c..9ab1792 100644
@@ -285,7 +285,7 @@ static int map_grant_pages(struct grant_map *map)
 
 	pr_debug("map %d+%d\n", map->index, map->count);
 	err = gnttab_map_refs(map->map_ops, use_ptemod ? map->kmap_ops : NULL,
-			map->pages, map->count);
+			      map->pages, map->count, 1);
 	if (err)
 		return err;
 
@@ -317,7 +317,7 @@ static int __unmap_grant_pages(struct grant_map *map, int offset, int pages)
 
 	err = gnttab_unmap_refs(map->unmap_ops + offset,
 			use_ptemod ? map->kmap_ops + offset : NULL, map->pages + offset,
-			pages);
+			pages, 1);
 	if (err)
 		return err;
 
drivers/xen/grant-table.c +39 -10

73 lines not on this page · raw message

include/xen/grant_table.h +12 -2
index 694dcaf..7fcb960 100644
@@ -183,12 +183,22 @@ unsigned int gnttab_max_grant_frames(void);
 
 #define gnttab_map_vaddr(map) ((void *)(map.host_virt_addr))
 
+/* override is used to add m2p override table entries when mapping the
+ * grant references.  Currently there are only two callers to this function,
+ * blkback and gntdev.  gntdev needs all grant mappings to have corresponding
+ * m2p override table entries but blkback currently doesn't.  This is because
+ * blkback can gracefully handle the case where m2p(p2m(pfn)) fails with
+ * foreign pfns.  If you cannot handle this correctly, you need to set
+ * override 1 when calling the map and unmap functions.
+ */
 int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		    struct gnttab_map_grant_ref *kmap_ops,
-		    struct page **pages, unsigned int count);
+		    struct page **pages, unsigned int count,
+		    int override);
 int gnttab_unmap_refs(struct gnttab_unmap_grant_ref *unmap_ops,
 		      struct gnttab_map_grant_ref *kunmap_ops,
-		      struct page **pages, unsigned int count);
+		      struct page **pages, unsigned int count,
+		      int override);
 
 /* Perform a batch of grant map/copy operations. Retry every batch slot
  * for which the hypervisor returns GNTST_eagain. This is typically due
Signature · 1 line
1.7.9.5
Reviewed-by on v1 1/1
#1 lore raw
Quoted 11 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

Quoted 4 lines

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

One comment for discussion below.

Quoted 3 lines

---
A backported version of this has been heavily tested but the testing against the latest Linux tree is light so far.

[...]

Quoted 18 lines
drivers/xen/grant-table.c +2 -1
index c4d2298..081be8d 100644
@@ -881,7 +881,8 @@ EXPORT_SYMBOL_GPL(gnttab_batch_copy);
 
 int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		    struct gnttab_map_grant_ref *kmap_ops,
-		    struct page **pages, unsigned int count)
+		    struct page **pages, unsigned int count,
+		    int override)
 {
 	int i, ret;
 	bool lazy = false;
@@ -918,10 +919,29 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		} else {
 			mfn = PFN_DOWN(map_ops[i].dev_bus_addr);
 		}

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 927

Quoted 10 lines
drivers/xen/grant-table.c +6 -4
-		ret = m2p_add_override(mfn, pages[i], kmap_ops ?
-				       &kmap_ops[i] : NULL);
-		if (ret)
-			return ret;
+
+		if (override) {
+			ret = m2p_add_override(mfn, pages[i], kmap_ops ?
+					       &kmap_ops[i] : NULL);
+			if (ret)
+				return ret;

The original code, which remains unmodified by this patch, is returning without calling arch_leave_lazy_mmu_mode() at the bottom of this function.

Quoted 4 lines
drivers/xen/grant-table.c +4 -0
+		} else {
+			unsigned long pfn, old_mfn;
+
+			pfn = page_to_pfn(pages[i]);

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 941

Quoted 10 lines
drivers/xen/grant-table.c +10 -0
+			old_mfn = pfn_to_mfn(pfn);
+
+			/* Save previous MFN in page private*/
+			WARN_ON(PagePrivate(pages[i]));
+			SetPagePrivate(pages[i]);
+			set_page_private(pages[i], old_mfn);
+
+			if (!set_phys_to_machine(pfn, FOREIGN_FRAME(mfn))) {
+				ret = -ENOMEM;
+				break;

The new path doesn't repeat this error. I think that the override path needs to be changed to do the same in a separate patch.

--msw

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 947

Quoted 6 lines
drivers/xen/grant-table.c +3 -0
+			}
+		}
+			
 	}
 
 	if (lazy)
                arch_leave_lazy_mmu_mode();

        return ret;
}
Reviewed-by on v1 1/1
#2 lore raw
In reply to Anthony Liguori
Quoted 11 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

Quoted 4 lines

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

One comment for discussion below.

Quoted 3 lines

---
A backported version of this has been heavily tested but the testing against the latest Linux tree is light so far.

[...]

Quoted 18 lines
drivers/xen/grant-table.c +2 -1
index c4d2298..081be8d 100644
@@ -881,7 +881,8 @@ EXPORT_SYMBOL_GPL(gnttab_batch_copy);
 
 int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		    struct gnttab_map_grant_ref *kmap_ops,
-		    struct page **pages, unsigned int count)
+		    struct page **pages, unsigned int count,
+		    int override)
 {
 	int i, ret;
 	bool lazy = false;
@@ -918,10 +919,29 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		} else {
 			mfn = PFN_DOWN(map_ops[i].dev_bus_addr);
 		}

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 927

Quoted 10 lines
drivers/xen/grant-table.c +6 -4
-		ret = m2p_add_override(mfn, pages[i], kmap_ops ?
-				       &kmap_ops[i] : NULL);
-		if (ret)
-			return ret;
+
+		if (override) {
+			ret = m2p_add_override(mfn, pages[i], kmap_ops ?
+					       &kmap_ops[i] : NULL);
+			if (ret)
+				return ret;

The original code, which remains unmodified by this patch, is returning without calling arch_leave_lazy_mmu_mode() at the bottom of this function.

Quoted 4 lines
drivers/xen/grant-table.c +4 -0
+		} else {
+			unsigned long pfn, old_mfn;
+
+			pfn = page_to_pfn(pages[i]);

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 941

Quoted 10 lines
drivers/xen/grant-table.c +10 -0
+			old_mfn = pfn_to_mfn(pfn);
+
+			/* Save previous MFN in page private*/
+			WARN_ON(PagePrivate(pages[i]));
+			SetPagePrivate(pages[i]);
+			set_page_private(pages[i], old_mfn);
+
+			if (!set_phys_to_machine(pfn, FOREIGN_FRAME(mfn))) {
+				ret = -ENOMEM;
+				break;

The new path doesn't repeat this error. I think that the override path needs to be changed to do the same in a separate patch.

--msw

drivers/xen/grant-table.c int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops, · line 947

Quoted 6 lines
drivers/xen/grant-table.c +3 -0
+			}
+		}
+			
 	}
 
 	if (lazy)
                arch_leave_lazy_mmu_mode();

        return ret;
}
#3 lore raw
In reply to Anthony Liguori
Quoted 6 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

Quoted 7 lines

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

I think it would be better if it was made clearer what m2p_add/remove_override() is for (i.e., allowing get_user_pages_fast() to work) so there isn't this confusion in the future. Please add some comments for this and move the calls into the gntdev driver.

In the future I would like to see the grant map/unmap done in m2p_add/remove_override() moved into gntdev as well, but this isn't a requirement for this performance fix.

As a prerequiste, the call to dma_mark_clean() in xen_swiotlb_unmap() needs to be removed and replaced with a comment explaining why this now differs from the generic implementation. i.e., the necessary phys_to_virt() will not work for foreign pages and dma_mark_clean() is a nop on all Xen supported architectures.

David

#4 lore raw
In reply to Anthony Liguori
Quoted 6 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

Quoted 7 lines

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

I think it would be better if it was made clearer what m2p_add/remove_override() is for (i.e., allowing get_user_pages_fast() to work) so there isn't this confusion in the future. Please add some comments for this and move the calls into the gntdev driver.

In the future I would like to see the grant map/unmap done in m2p_add/remove_override() moved into gntdev as well, but this isn't a requirement for this performance fix.

As a prerequiste, the call to dma_mark_clean() in xen_swiotlb_unmap() needs to be removed and replaced with a comment explaining why this now differs from the generic implementation. i.e., the necessary phys_to_virt() will not work for foreign pages and dma_mark_clean() is a nop on all Xen supported architectures.

David

#5 lore raw
Quoted 12 lines · David Vrabel
Quoted 11 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

Quoted 7 lines
Quoted 2 lines

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

I think it would be better if it was made clearer what m2p_add/remove_override() is for (i.e., allowing get_user_pages_fast() to work) so there isn't this confusion in the future. Please add some comments for this and move the calls into the gntdev driver.

Even though get_user_pages_fast is obviously the most important use case for the m2p_override, it is not the only one, see below.

Quoted 4 lines

As a prerequiste, the call to dma_mark_clean() in xen_swiotlb_unmap() needs to be removed and replaced with a comment explaining why this now differs from the generic implementation. i.e., the necessary phys_to_virt() will not work for foreign pages

Actually it is machine_to_phys that is not going to work and only on x86. machine_to_phys is going to be fine for foreign pages on ARM.

Let's be clear about this: without the m2p_override on x86 xen_swiotlb_unmap is going to fail and that undeniably is a mistake. However it is a mistake that happens not to cause any problems today.

In fact the following comment is not right:

+/* override is used to add m2p override table entries when mapping the
+ * grant references. Currently there are only two callers to this function,
+ * blkback and gntdev. gntdev needs all grant mappings to have corresponding
+ * m2p override table entries but blkback currently doesn't. This is because
+ * blkback can gracefully handle the case where m2p(p2m(pfn)) fails with
+ * foreign pfns. If you cannot handle this correctly, you need to set
+ * override 1 when calling the map and unmap functions.
+ */

I would add the following comment to xen_unmap_single instead:

/* On x86 xen_bus_to_phys is going to fail for foreign pages unless they
* have been added to the m2p_override. Only gntdev currently adds them to
* the m2p_override, while blkback does not. As a consequence any foreign
* pages mapped by blkback and used for DMA are not going to be unmapped
* correctly here. Fortunately this is not a problem because this function
* is actually a nop for non-swiotlb pages on x86.
*/

Quoted 1 line

and dma_mark_clean() is a nop on all Xen supported architectures.

That's true, it should be removed. It is just confusing.

#6 lore raw
In reply to David Vrabel
Quoted 12 lines · David Vrabel
Quoted 11 lines · Anthony Liguori

From: Anthony Liguori <[email protected]>

Commit 5dc03639 switched blkback to also add m2p override entries when mapping grant pages but history seems to have forgotten why this is useful if it ever was.

The blkback driver does not need m2p override entries to exist and there is significant overhead due to the locking in the m2p override table. We see about a 10% improvement in IOP rate with this patch applied running FIO in the guest.

Quoted 7 lines
Quoted 2 lines

See http://article.gmane.org/gmane.linux.kernel/1590932 for a full analysis of current users.

I think it would be better if it was made clearer what m2p_add/remove_override() is for (i.e., allowing get_user_pages_fast() to work) so there isn't this confusion in the future. Please add some comments for this and move the calls into the gntdev driver.

Even though get_user_pages_fast is obviously the most important use case for the m2p_override, it is not the only one, see below.

Quoted 4 lines

As a prerequiste, the call to dma_mark_clean() in xen_swiotlb_unmap() needs to be removed and replaced with a comment explaining why this now differs from the generic implementation. i.e., the necessary phys_to_virt() will not work for foreign pages

Actually it is machine_to_phys that is not going to work and only on x86. machine_to_phys is going to be fine for foreign pages on ARM.

Let's be clear about this: without the m2p_override on x86 xen_swiotlb_unmap is going to fail and that undeniably is a mistake. However it is a mistake that happens not to cause any problems today.

In fact the following comment is not right:

+/* override is used to add m2p override table entries when mapping the
+ * grant references. Currently there are only two callers to this function,
+ * blkback and gntdev. gntdev needs all grant mappings to have corresponding
+ * m2p override table entries but blkback currently doesn't. This is because
+ * blkback can gracefully handle the case where m2p(p2m(pfn)) fails with
+ * foreign pfns. If you cannot handle this correctly, you need to set
+ * override 1 when calling the map and unmap functions.
+ */

I would add the following comment to xen_unmap_single instead:

/* On x86 xen_bus_to_phys is going to fail for foreign pages unless they
* have been added to the m2p_override. Only gntdev currently adds them to
* the m2p_override, while blkback does not. As a consequence any foreign
* pages mapped by blkback and used for DMA are not going to be unmapped
* correctly here. Fortunately this is not a problem because this function
* is actually a nop for non-swiotlb pages on x86.
*/

Quoted 1 line

and dma_mark_clean() is a nop on all Xen supported architectures.

That's true, it should be removed. It is just confusing.