Skip to contentSkip to conversation

gnttab: Improve scaleability

PATCH Superseded Christoph Egger 3 participants 10 messages Dec 2, 2014

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover gnttab: Improve scaleability

This patch series changes the grant table locking to a more fain grained locking protocol. The result is a performance boost measured with blkfront/blkback. Document the locking protocol.

[PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant [PATCH 2/2] gnttab: refactor locking for scalability

#1 lore raw

PATCH 1/2 gnttab: Introduce rwlock to protect updates to grant table state

Rename lock to maptrack_lock and use it to protect maptrack state.

The new rwlock is used to prevent readers from accessing inconsistent grant table state such as current version, partially initialized active table pages, etc.

[chegger: ported to xen-staging, split into multiple commits]
Cc Jan Beulich <[email protected]>
Cc Keir Fraser <[email protected]>
---
 docs/misc/grant-tables.txt    |   28 ++++++++-
 xen/arch/x86/mm.c             |    4 +-
 xen/common/grant_table.c      |  135 ++++++++++++++++++++++++-----------------
 xen/include/xen/grant_table.h |    9 ++-
 4 files changed, 115 insertions(+), 61 deletions(-)
docs/misc/grant-tables.txt +27 -1
index 19db4ec..c9ae8f2 100644
@@ -74,7 +74,33 @@ is complete.
  matching map track entry is then removed, as if unmap had been invoked.
  These are not used by the transfer mechanism.
   map->domid         : owner of the mapped frame
-  map->ref_and_flags : grant reference, ro/rw, mapped for host or device access
+  map->ref           : grant reference
+  map->flags         : ro/rw, mapped for host or device access
+
+********************************************************************************
+ Locking
+ ~~~~~~~
+ Xen uses several locks to serialize access to the internal grant table state.
+
+  grant_table->lock          : rwlock used to prevent readers from accessing
+                               inconsistent grant table state such as current
+                               version, partially initialized active table pages,
+                               etc.
+  grant_table->maptrack_lock : spinlock used to protect the maptrack state
+
+ The primary lock for the grant table is a read/write spinlock. All
+ functions that access members of struct grant_table must acquire a
+ read lock around critical sections. Any modification to the members
+ of struct grant_table (e.g., nr_status_frames, nr_grant_frames,
+ active frames, etc.) must only be made if the write lock is
+ held. These elements are read-mostly, and read critical sections can
+ be large, which makes a rwlock a good choice.
+
+ The maptrack state is protected by its own spinlock. Any access (read
+ or write) of struct grant_table members that have a "maptrack_"
+ prefix must be made while holding the maptrack lock. The maptrack
+ state can be rapidly modified under some workloads, and the critical
+ sections are very small, thus we use a spinlock to protect them.
 
 ********************************************************************************
 
xen/arch/x86/mm.c +2 -2
index 522c43d..37c13b1 100644
@@ -4565,7 +4565,7 @@ int xenmem_add_to_physmap_one(
                 mfn = virt_to_mfn(d->shared_info);
             break;
         case XENMAPSPACE_grant_table:
-            spin_lock(&d->grant_table->lock);
+            write_lock(&d->grant_table->lock);
 
             if ( d->grant_table->gt_version == 0 )
                 d->grant_table->gt_version = 1;
@@ -4587,7 +4587,7 @@ int xenmem_add_to_physmap_one(
                     mfn = virt_to_mfn(d->grant_table->shared_raw[idx]);
             }
 
-            spin_unlock(&d->grant_table->lock);
+            write_unlock(&d->grant_table->lock);
             break;
         case XENMAPSPACE_gmfn_range:
         case XENMAPSPACE_gmfn:
xen/common/grant_table.c +80 -55

461 lines not on this page · raw message

xen/include/xen/grant_table.h +6 -3
index 32f5786..ca49b41 100644
@@ -82,8 +82,11 @@ struct grant_table {
     struct grant_mapping **maptrack;
     unsigned int          maptrack_head;
     unsigned int          maptrack_limit;
-    /* Lock protecting updates to active and shared grant tables. */
-    spinlock_t            lock;
+    /* Lock protecting the maptrack page list, head, and limit */
+    spinlock_t            maptrack_lock;
+    /* Lock protecting updates to grant table state (version, active
+       entry list, etc.) */
+    rwlock_t              lock;
     /* The defined versions are 1 and 2.  Set to 0 if we don't know
        what version to use yet. */
     unsigned              gt_version;
@@ -101,7 +104,7 @@ gnttab_release_mappings(
     struct domain *d);
 
 /* Increase the size of a domain's grant table.
- * Caller must hold d's grant table lock.
+ * Caller must hold d's grant table write lock.
  */
 int
 gnttab_grow_table(struct domain *d, unsigned int req_nr_frames);
Signature · 1 line
1.7.9.5
#2 lore raw

Re: [PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant table state

Quoted 6 lines · Christoph Egger

Rename lock to maptrack_lock and use it to protect maptrack state.

The new rwlock is used to prevent readers from accessing inconsistent grant table state such as current version, partially initialized active table pages, etc.

I would suggest phrasing this slightly differently, as it isn't a simple rename.

What you are doing is taking the existing grant table lock, splitting it in two to separate the grant locking from the maptrack locking, and changing the resulting grant lock to be a rwlock.

With this noted, it would certainly be easier to review if this patch was split in two; one patch to split the spinlocks and one patch to change the grant lock from a spinlock to a rwlock.

Quoted 13 lines
xen/common/grant_table.c +3 -2
index 8fba923..018b5b6 100644
@@ -2501,8 +2510,19 @@ gnttab_swap_grant_ref(XEN_GUEST_HANDLE_PARAM(gnttab_swap_grant_ref_t) uop,
             return i;
         if ( unlikely(__copy_from_guest(&op, uop, 1)) )
             return -EFAULT;
-        op.status = __gnttab_swap_grant_ref(op.ref_a, op.ref_b);
-        if ( unlikely(__copy_field_to_guest(uop, &op, status)) )
+        if ( unlikely(op.ref_a == op.ref_b) )
+        {
+            /* noop, so avoid acquiring the same active entry

xen/common/grant_table.c gnttab_swap_grant_ref(XEN_GUEST_HANDLE_PARAM(gnttab_swap_grant_ref_t) uop, · line 2525

Quoted 10 lines
xen/common/grant_table.c +10 -0
+             * twice in __gnttab_swap_grant_ref(), which would
+             * case a deadlock.
+             */
+            op.status = GNTST_okay;
+        }
+        else
+        {
+            op.status = __gnttab_swap_grant_ref(op.ref_a, op.ref_b);
+        }
+        if ( unlikely(__copy_to_guest_offset(uop, i, &op, 1)) )

I believe this comment only applies to the changes in active locking introduced in the subsequent patch?

Either way, I think the a == b check should be in __gnttab_swap_grant_ref() make any caller safe, not just the this one.

~Andrew

#3 lore raw

Re: [PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant table state

In reply to Christoph Egger

Hi Christoph,

Quoted 14 lines · Christoph Egger
xen/arch/x86/mm.c +1 -1
index 522c43d..37c13b1 100644
@@ -4565,7 +4565,7 @@ int xenmem_add_to_physmap_one(
                 mfn = virt_to_mfn(d->shared_info);
             break;
         case XENMAPSPACE_grant_table:
-            spin_lock(&d->grant_table->lock);
+            write_lock(&d->grant_table->lock);
 
             if ( d->grant_table->gt_version == 0 )
                 d->grant_table->gt_version = 1;

xen/arch/x86/mm.c int xenmem_add_to_physmap_one( · line 4593

Quoted 8 lines
xen/arch/x86/mm.c +1 -1
@@ -4587,7 +4587,7 @@ int xenmem_add_to_physmap_one(
                     mfn = virt_to_mfn(d->grant_table->shared_raw[idx]);
             }
 
-            spin_unlock(&d->grant_table->lock);
+            write_unlock(&d->grant_table->lock);
             break;
         case XENMAPSPACE_gmfn_range:
         case XENMAPSPACE_gmfn:

You forgot to modify the ARM bits which is using the spinlock. See arch/arm/mm.c

Regards,

Signature · 1 line
Julien Grall
#4 lore raw

Re: [PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant table state

Quoted 20 lines · Julien Grall

Hi Christoph,

Quoted 17 lines · Christoph Egger
xen/arch/x86/mm.c +1 -1
index 522c43d..37c13b1 100644
@@ -4565,7 +4565,7 @@ int xenmem_add_to_physmap_one(
                 mfn = virt_to_mfn(d->shared_info);
             break;
         case XENMAPSPACE_grant_table:
-            spin_lock(&d->grant_table->lock);
+            write_lock(&d->grant_table->lock);
 
             if ( d->grant_table->gt_version == 0 )
                 d->grant_table->gt_version = 1;
@@ -4587,7 +4587,7 @@ int xenmem_add_to_physmap_one(
                     mfn = virt_to_mfn(d->grant_table->shared_raw[idx]);
             }
 
Quoted 8 lines

xen/arch/x86/mm.c int xenmem_add_to_physmap_one( · line 4593

Quoted 5 lines
xen/arch/x86/mm.c +1 -1
-            spin_unlock(&d->grant_table->lock);
+            write_unlock(&d->grant_table->lock);
             break;
         case XENMAPSPACE_gmfn_range:
         case XENMAPSPACE_gmfn:

You forgot to modify the ARM bits which is using the spinlock. See arch/arm/mm.c

I can do the change. But I don't have ARM hardware nor the build infrastructure. I need your help with compiling and testing on/for ARM. Alternatively you can send me a patch I can add to or squash into my patch series.

Christoph

#5 lore raw

Re: [PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant table state

Hi,

Quoted 30 lines · Christoph
Quoted 20 lines · Julien Grall

Hi Christoph,

Quoted 17 lines · Christoph Egger
xen/arch/x86/mm.c +1 -1
index 522c43d..37c13b1 100644
@@ -4565,7 +4565,7 @@ int xenmem_add_to_physmap_one(
                 mfn = virt_to_mfn(d->shared_info);
             break;
         case XENMAPSPACE_grant_table:
-            spin_lock(&d->grant_table->lock);
+            write_lock(&d->grant_table->lock);
 
             if ( d->grant_table->gt_version == 0 )
                 d->grant_table->gt_version = 1;
@@ -4587,7 +4587,7 @@ int xenmem_add_to_physmap_one(
                     mfn = virt_to_mfn(d->grant_table->shared_raw[idx]);
             }
 
Quoted 8 lines

xen/arch/x86/mm.c int xenmem_add_to_physmap_one( · line 4593

Quoted 5 lines
xen/arch/x86/mm.c +1 -1
-            spin_unlock(&d->grant_table->lock);
+            write_unlock(&d->grant_table->lock);
             break;
         case XENMAPSPACE_gmfn_range:
         case XENMAPSPACE_gmfn:

You forgot to modify the ARM bits which is using the spinlock. See arch/arm/mm.c

Quoted 4 lines

I can do the change. But I don't have ARM hardware nor the build infrastructure. I need your help with compiling and testing on/for ARM. Alternatively you can send me a patch I can add to or squash into my patch series.

I agree that testing the ARM part may require a bit of setup. But you can easily build testing. only a cross-compiler for ARM is required for this step. See:

http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Building_Xen_on_ARM

Regards,

Signature · 1 line
Julien Grall
#6 lore raw

Re: [PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant table state

Quoted 38 lines · Julien Grall

Hi,

Quoted 30 lines · Christoph
Quoted 20 lines · Julien Grall

Hi Christoph,

Quoted 17 lines · Christoph Egger
xen/arch/x86/mm.c +1 -1
index 522c43d..37c13b1 100644
@@ -4565,7 +4565,7 @@ int xenmem_add_to_physmap_one(
                 mfn = virt_to_mfn(d->shared_info);
             break;
         case XENMAPSPACE_grant_table:
-            spin_lock(&d->grant_table->lock);
+            write_lock(&d->grant_table->lock);
 
             if ( d->grant_table->gt_version == 0 )
                 d->grant_table->gt_version = 1;
@@ -4587,7 +4587,7 @@ int xenmem_add_to_physmap_one(
                     mfn = virt_to_mfn(d->grant_table->shared_raw[idx]);
             }
 
Quoted 8 lines

xen/arch/x86/mm.c int xenmem_add_to_physmap_one( · line 4593

Quoted 5 lines
xen/arch/x86/mm.c +1 -1
-            spin_unlock(&d->grant_table->lock);
+            write_unlock(&d->grant_table->lock);
             break;
         case XENMAPSPACE_gmfn_range:
         case XENMAPSPACE_gmfn:

You forgot to modify the ARM bits which is using the spinlock. See arch/arm/mm.c

Quoted 4 lines

I can do the change. But I don't have ARM hardware nor the build infrastructure. I need your help with compiling and testing on/for ARM. Alternatively you can send me a patch I can add to or squash into my patch series.

Quoted 5 lines

I agree that testing the ARM part may require a bit of setup. But you can easily build testing. only a cross-compiler for ARM is required for this step. See:

http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Building_Xen_on_ARM

Thank you for this pointer.

I just compared xenmem_add_to_physmap_one() x86 and arm versions. The real architectural functions are virt_to_mfn(), mfn_to_page() and friends. Everything else in this function is just common code and can be moved into xen/common/ .

Christoph

#7 lore raw

PATCH 2/2 gnttab: refactor locking for scalability

In reply to Christoph Egger

From: Matt Wilson <[email protected]>

This patch refactors grant table locking. It splits the previous single spinlock per grant table into multiple locks. The heavily modified components of the grant table (the maptrack state and the active entries) are now protected by their own spinlocks. The remaining elements of the grant table are read-mostly, so the main grant table lock is modified to be a rwlock to improve concurrency.

Workloads with high grant table operation rates, especially map/unmap operations as used by blkback/blkfront when persistent grants are not supported, show marked improvement with these changes. A domU with 24 VBDs in a streaming 2M write workload achieved 1,400 MB/sec before this change. Performance more than doubles with this patch, reaching 3,000 MB/sec before tuning and 3,600 MB/sec after adjusting event channel vCPU bindings.

[chegger: ported to xen-staging, split into multiple commits]
Cc Anthony Liguori <[email protected]>
Cc Jan Beulich <[email protected]>
Cc Keir Fraser <[email protected]>
---
 docs/misc/grant-tables.txt |   21 +++++
 xen/common/grant_table.c   |  213 ++++++++++++++++++++++++++++----------------
 2 files changed, 157 insertions(+), 77 deletions(-)
docs/misc/grant-tables.txt +21 -0
index c9ae8f2..1ada018 100644
@@ -63,6 +63,7 @@ is complete.
   act->domid : remote domain being granted rights
   act->frame : machine frame being granted
   act->pin   : used to hold reference counts
+  act->lock  : spinlock used to serialize access to active entry state
 
  Map tracking
  ~~~~~~~~~~~~
@@ -87,6 +88,8 @@ is complete.
                                version, partially initialized active table pages,
                                etc.
   grant_table->maptrack_lock : spinlock used to protect the maptrack state
+  active_grant_entry->lock   : spinlock used to serialize modifications to
+                               active entries
 
  The primary lock for the grant table is a read/write spinlock. All
  functions that access members of struct grant_table must acquire a
@@ -102,6 +105,24 @@ is complete.
  state can be rapidly modified under some workloads, and the critical
  sections are very small, thus we use a spinlock to protect them.
 
+ Active entries are obtained by calling active_entry_acquire(gt, ref).
+ This function returns a pointer to the active entry after locking its
+ spinlock. The caller must hold the rwlock for the gt in question
+ before calling active_entry_acquire(). This is because the grant
+ table can be dynamically extended via gnttab_grow_table() while a
+ domain is running and must be fully initialized. Once all access to
+ the active entry is complete, release the lock by calling
+ active_entry_release(act).
+
+ Summary of rules for locking:
+  active_entry_acquire() and active_entry_release() can only be
+  called when holding the relevant grant table's lock. I.e.:
+    read_lock(&gt->lock);
+    act = active_entry_acquire(gt, ref);
+    ...
+    active_entry_release(act);
+    read_unlock(&gt->lock);
+
 ********************************************************************************
 
  Granting a foreign domain access to frames
xen/common/grant_table.c +136 -77

581 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#8 lore raw
In reply to Christoph Egger
Quoted 7 lines · Christoph Egger

This patch series changes the grant table locking to a more fain grained locking protocol. The result is a performance boost measured with blkfront/blkback. Document the locking protocol.

[PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant [PATCH 2/2] gnttab: refactor locking for scalability

XenServer has been using the original version of this patch (forward ported a little), and it makes significant improvements to our scalability.

I highly recommend this for inclusion, although for 4.6 at this point.

~Andrew

#9 lore raw
Quoted 10 lines · Andrew Cooper
Quoted 7 lines · Christoph Egger

This patch series changes the grant table locking to a more fain grained locking protocol. The result is a performance boost measured with blkfront/blkback. Document the locking protocol.

[PATCH 1/2] gnttab: Introduce rwlock to protect updates to grant [PATCH 2/2] gnttab: refactor locking for scalability

XenServer has been using the original version of this patch

Maybe the one sent by Matt Wilson some time ago?

Quoted 1 line

(forward ported a little),

This version is against staging tree.

Quoted 1 line

and it makes significant improvements to our scalability.

Thanks for confirmation.

Quoted 1 line

I highly recommend this for inclusion, although for 4.6 at this point.

I am fine with this. I would be happy to see a backport to xen 4.5 and 4.4 then.

Christoph