gnttab: Improve scaleability
v5 cover gnttab: Improve scaleability
This patch series changes the grant table locking to a more fain grained locking protocol. The result is a performance boost measured with blkfront/blkback. Document the locking protocol.
v5:
* Addressed locking issue pointed out by Jan Beulich
* Fixed git rebase merge issue introduced in v4 (acquiring locking twice)
* Change for ()-loop in grant_map_exists
* Coding style fixes
v4:
* Coding style nits from Jan Beulich
* Fixup read locks pointed out by Jan Beulich
* renamed double_gt_(un)lock to double_maptrack_(un)lock per request from Jan Beulich
* Addressed ASSERT()'s from Jan Beulich
v3:
* Addressed gnttab_swap_grant_ref() comment from Andrew Cooper
v2:
* Add arm part per request from Julien Grall
Christoph Egger (1):
gnttab: Introduce rwlock to protect updates to grant table state
Matt Wilson (1):
gnttab: refactor locking for scalability
docs/misc/grant-tables.txt | 49 ++++++- xen/arch/arm/mm.c | 4 +- xen/arch/x86/mm.c | 4 +- xen/common/grant_table.c | 321 +++++++++++++++++++++++++---------------- xen/include/xen/grant_table.h | 9 +- 5 files changed, 258 insertions(+), 129 deletions(-)
Signature · 1 line
1.7.9.5
PATCH v5 1/2 gnttab: Introduce rwlock to protect updates to grant table state
Split grant table lock into two separate locks. One to protect maptrack state and change the other into a rwlock.
The rwlock is used to prevent readers from accessing inconsistent grant table state such as current version, partially initialized active table pages, etc.
v5:
* Addressed locking issue pointed out by Jan Beulich
* Fixed git rebase merge issue introduced in v4 (acquiring locking twice)
* Coding style fixes
v4:
* Coding style nits from Jan Beulich
* Fixup read locks pointed out by Jan Beulich
* renamed double_gt_(un)lock to double_maptrack_(un)lock per request from Jan Beulich
v3:
* Addressed gnttab_swap_grant_ref() comment from Andrew Cooper
v2:
* Add arm part per request from Julien Grall
Adjust locking
--- docs/misc/grant-tables.txt | 28 +++++++- xen/arch/arm/mm.c | 4 +- xen/arch/x86/mm.c | 4 +- xen/common/grant_table.c | 146 ++++++++++++++++++++++++----------------- xen/include/xen/grant_table.h | 17 +++-- 5 files changed, 127 insertions(+), 72 deletions(-)
docs/misc/grant-tables.txt +27 -1
@@ -74,7 +74,33 @@ is complete. matching map track entry is then removed, as if unmap had been invoked. These are not used by the transfer mechanism. map->domid : owner of the mapped frame - map->ref_and_flags : grant reference, ro/rw, mapped for host or device access + map->ref : grant reference + map->flags : ro/rw, mapped for host or device access + +******************************************************************************** + Locking + ~~~~~~~ + Xen uses several locks to serialize access to the internal grant table state. + + grant_table->lock : rwlock used to prevent readers from accessing + inconsistent grant table state such as current + version, partially initialized active table pages, + etc. + grant_table->maptrack_lock : spinlock used to protect the maptrack state + + The primary lock for the grant table is a read/write spinlock. All + functions that access members of struct grant_table must acquire a + read lock around critical sections. Any modification to the members + of struct grant_table (e.g., nr_status_frames, nr_grant_frames, + active frames, etc.) must only be made if the write lock is + held. These elements are read-mostly, and read critical sections can + be large, which makes a rwlock a good choice. + + The maptrack state is protected by its own spinlock. Any access (read + or write) of struct grant_table members that have a "maptrack_" + prefix must be made while holding the maptrack lock. The maptrack + state can be rapidly modified under some workloads, and the critical + sections are very small, thus we use a spinlock to protect them. ********************************************************************************
xen/arch/arm/mm.c +2 -2
@@ -1037,7 +1037,7 @@ int xenmem_add_to_physmap_one(switch ( space ) {case XENMAPSPACE_grant_table: -spin_lock (&d->grant_table->lock); +write_lock (&d->grant_table->lock);if ( d->grant_table->gt_version ==0 ) d->grant_table->gt_version =1 ;
@@ -1067,7 +1067,7 @@ int xenmem_add_to_physmap_one( t = p2m_ram_rw; -spin_unlock (&d->grant_table->lock); +write_unlock (&d->grant_table->lock);break ;case XENMAPSPACE_shared_info:if ( idx !=0 )
xen/arch/x86/mm.c +2 -2
@@ -4569,7 +4569,7 @@ int xenmem_add_to_physmap_one( mfn =virt_to_mfn (d->shared_info);break ;case XENMAPSPACE_grant_table: -spin_lock (&d->grant_table->lock); +write_lock (&d->grant_table->lock);if ( d->grant_table->gt_version ==0 ) d->grant_table->gt_version =1 ;
@@ -4591,7 +4591,7 @@ int xenmem_add_to_physmap_one( mfn =virt_to_mfn (d->grant_table->shared_raw[idx]); } -spin_unlock (&d->grant_table->lock); +write_unlock (&d->grant_table->lock);break ;case XENMAPSPACE_gmfn_range:case XENMAPSPACE_gmfn:
xen/common/grant_table.c +84 -62
476 lines not on this page · raw message
xen/include/xen/grant_table.h +12 -5
@@ -82,10 +82,17 @@ struct grant_table {struct grant_mapping **maptrack;unsigned int maptrack_head;unsigned int maptrack_limit; -/* Lock protecting updates to active and shared grant tables. */ -spinlock_t lock; -/* The defined versions are 1 and 2. Set to 0 if we don't know -what version to use yet. */ +/* Lock protecting the maptrack page list, head, and limit */ +spinlock_t maptrack_lock; +/* +* Lock protecting updates to grant table state (version, active +* entry list, etc.) +*/ +rwlock_t lock; +/* +* The defined versions are 1 and 2. Set to 0 if we don't know +* what version to use yet. +*/ unsigned gt_version; };
@@ -101,7 +108,7 @@ gnttab_release_mappings( struct domain *d);/* Increase the size of a domain's grant table. -* Caller must hold d's grant table lock. +* Caller must hold d's grant table write lock. */ int gnttab_grow_table (struct domain *d,unsigned int req_nr_frames);
Signature · 1 line
1.7.9.5
Re: [PATCH v5 1/2] gnttab: Introduce rwlock to protect updates to grant table state
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 10.02.15 at 13:51, <[email protected]> wrote:
xen/common/grant_table.c +1
-3@@ -629,7 +629,7 @@ __gnttab_map_grant_ref( } rgt = rd->grant_table; -spin_lock (&rgt->lock); +read_lock (&rgt->lock);if ( rgt->gt_version ==0 ) PIN_FAIL(unlock_out, GNTST_general_error,@@ -702,8 +702,6 @@ __gnttab_map_grant_ref( cache_flags = (shah->flags & (GTF_PAT |GTF_PWT |GTF_PCD ) ); -spin_unlock (&rgt->lock); -/* pg may be set, with a refcount included, from __get_paged_frame */ if ( !pg ) {
xen/common/grant_table.c __gnttab_map_grant_ref( · line 776
Quoted 1 line
xen/common/grant_table.c +0
-0@@ -778,7 +776,7 @@ __gnttab_map_grant_ref( goto undo_out;
With the unlock above dropped, this and other similar goto-s now end up acquiring the lock a second time, but then dropping it only once. As said before (on a different piece of code) altering the regions which get locked should be made explicit in the patch description, allowing - if not spotted during review - later bug fixing to be done with proper understanding of the intentions.
xen/common/grant_table.c __gnttab_map_grant_ref( · line 779
Quoted 4 lines
xen/common/grant_table.c +1
-1} -double_gt_lock (lgt, rgt); +double_maptrack_lock (lgt, rgt);
Looking at this nested locking I find that you _still_ don't say anything about lock hierarchy in the doc change.
xen/common/grant_table.c __acquire_grant_for_copy( · line 1987
Quoted 11 lines
xen/common/grant_table.c +6
-1@@ -1971,17 +1979,23 @@ __acquire_grant_for_copy(PIN_FAIL (unlock_out_clear, GNTST_general_error,"transitive grant referenced bad domain %d\n" , trans_domid); -spin_unlock (&rgt->lock); + +/* +* __acquire_grant_for_copy() could take the read lock on +* the right table (if rd == td), so we have to drop the +* lock here and reacquire +*/
Almost - there's still a full stop missing here.
Quoted 12 lines
xen/common/grant_table.c +3
-1@@ -3129,7 +3147,9 @@ grant_table_destroy(if ( t ==NULL )return ; - + +write_lock (&t->lock); +for ( i =0 ; i <nr_grant_frames (t); i++ )free_xenheap_page (t->shared_raw[i]);xfree (t->shared_raw);
xen/common/grant_table.c grant_table_destroy( · line 3169
Quoted 4 lines
xen/common/grant_table.c +1
-0@@ -3146,6 +3166,8 @@ grant_table_destroy(free_xenheap_page (t->status[i]);xfree (t->status); +write_unlock (&t->lock);
Do we really need the lock here? If so (as also said before) this ought to be a separate fix (needing backporting).
Quoted 7 lines
xen/include/xen/grant_table.h +0
-1@@ -82,10 +82,17 @@ struct grant_table {struct grant_mapping **maptrack;unsigned int maptrack_head;unsigned int maptrack_limit; -/* Lock protecting updates to active and shared grant tables. */
xen/include/xen/grant_table.h struct grant_table { · line 91
Quoted 10 lines
xen/include/xen/grant_table.h +7
-3-spinlock_t lock; -/* The defined versions are 1 and 2. Set to 0 if we don't know -what version to use yet. */ +/* Lock protecting the maptrack page list, head, and limit */ +spinlock_t maptrack_lock; +/* +* Lock protecting updates to grant table state (version, active +* entry list, etc.) +*/ +rwlock_t lock;
Repeating the recently raised similar question (in another context):
Is it really a good idea to have two locks on the same cache line?
Jan
PATCH v5 2/2 gnttab: refactor locking for scalability
In reply to Christoph EggerFrom: Matt Wilson <[email protected]>
This patch refactors grant table locking. It splits the previous single spinlock per grant table into multiple locks. The heavily modified components of the grant table (the maptrack state and the active entries) are now protected by their own spinlocks. The remaining elements of the grant table are read-mostly, so the main grant table lock is modified to be a rwlock to improve concurrency.
Workloads with high grant table operation rates, especially map/unmap operations as used by blkback/blkfront when persistent grants are not supported, show marked improvement with these changes. A domU with 24 VBDs in a streaming 2M write workload achieved 1,400 MB/sec before this change. Performance more than doubles with this patch, reaching 3,000 MB/sec before tuning and 3,600 MB/sec after adjusting event channel vCPU bindings.
v5:
* Fixed git rebase merge issue introduced in v4 (acquiring locking twice)
* Change for ()-loop in grant_map_exists
* Coding style fixes
v4:
* Addressed ASSERT()'s from Jan Beulich
* Addressed locking issues in unmap_common pointed out by Jan Beulich
v3:
* Addressed gnttab_swap_grant_ref() comment from Andrew Cooper
--- docs/misc/grant-tables.txt | 21 +++++ xen/common/grant_table.c | 215 +++++++++++++++++++++++++++----------------- 2 files changed, 155 insertions(+), 81 deletions(-)
docs/misc/grant-tables.txt +21 -0
@@ -63,6 +63,7 @@ is complete. act->domid : remote domain being granted rights act->frame : machine frame being granted act->pin : used to hold reference counts + act->lock : spinlock used to serialize access to active entry state Map tracking ~~~~~~~~~~~~
@@ -87,6 +88,8 @@ is complete. version, partially initialized active table pages, etc. grant_table->maptrack_lock : spinlock used to protect the maptrack state + active_grant_entry->lock : spinlock used to serialize modifications to + active entries The primary lock for the grant table is a read/write spinlock. All functions that access members of struct grant_table must acquire a
@@ -102,6 +105,24 @@ is complete. state can be rapidly modified under some workloads, and the critical sections are very small, thus we use a spinlock to protect them. + Active entries are obtained by calling active_entry_acquire(gt, ref). + This function returns a pointer to the active entry after locking its + spinlock. The caller must hold the rwlock for the gt in question + before calling active_entry_acquire(). This is because the grant + table can be dynamically extended via gnttab_grow_table() while a + domain is running and must be fully initialized. Once all access to + the active entry is complete, release the lock by calling + active_entry_release(act). + + Summary of rules for locking: + active_entry_acquire() and active_entry_release() can only be + called when holding the relevant grant table's lock. I.e.: + read_lock(>->lock); + act = active_entry_acquire(gt, ref); + ... + active_entry_release(act); + read_unlock(>->lock); + ******************************************************************************** Granting a foreign domain access to frames
xen/common/grant_table.c +134 -81
558 lines not on this page · raw message
Signature · 1 line
1.7.9.5
Re: [PATCH v5 2/2] gnttab: refactor locking for scalability
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 10.02.15 at 13:51, <[email protected]> wrote:
xen/common/grant_table.c +0
-0@@ -188,6 +191,28 @@ nr_active_grant_frames(struct grant_table *gt)return num_act_frames_from_sha_frames (nr_grant_frames (gt)); }
xen/common/grant_table.c nr_active_grant_frames(struct grant_table *gt) · line 202
Quoted 10 lines
xen/common/grant_table.c +9
-0+static inline struct active_grant_entry * +active_entry_acquire (struct grant_table *t,grant_ref_t e) +{ +struct active_grant_entry *act; + +/* +* not perfect, but better than nothing for a debug build +* sanity check +*/
Better, but still not matching ./CODING_STYLE.
xen/common/grant_table.c get_maptrack_handle( · line 324
Quoted 10 lines
xen/common/grant_table.c +4
-1@@ -316,7 +317,10 @@ get_maptrack_handle(return handle; } -/* Number of grant table entries. Caller must hold d's grant table lock. */ +/* +* Number of grant table entries. Caller must hold d's grant table +* read lock. +*/ static unsigned int nr_grant_entries (struct grant_table *gt)
Doesn't this belong in patch 1?
Quoted 19 lines
xen/common/grant_table.c +3
-4@@ -505,26 +509,23 @@ static int grant_map_exists(const struct domain *ld,unsigned long mfn, unsigned int *ref_count) { - const struct active_grant_entry *act; + struct active_grant_entry *act;unsigned int ref, max_iter;ASSERT (rw_is_locked (&rgt->lock)); max_iter =min (*ref_count + (1 <<GNTTABOP_CONTINUATION_ARG_SHIFT ),nr_grant_entries (rgt)); -for ( ref = *ref_count; ref < max_iter; ref++ ) +for ( ref = *ref_count; ref < max_iter;active_entry_release (act), ref++ ) { - act = &active_entry (rgt, ref); + act =active_entry_acquire (rgt, ref); -if ( !act->pin )
xen/common/grant_table.c static int grant_map_exists(const struct domain *ld, · line 526
Quoted 10 lines
xen/common/grant_table.c +3
-6-continue ; - -if ( act->domid != ld->domain_id ) -continue ; - -if ( act->frame != mfn ) +if ( !act->pin || + act->domid != ld->domain_id || + act->frame != mfn )continue ;
I don't mind this consolidation, but it does grow the patch size without real need.
xen/common/grant_table.c static void mapcount( · line 580
Quoted 13 lines
xen/common/grant_table.c +4
-1for ( handle =0 ; handle < lgt->maptrack_limit; handle++ ) { +struct active_grant_entry *act; + map = &maptrack_entry (lgt, handle);if ( !(map->flags & (GNTMAP_device_map|GNTMAP_host_map)) || map->domid != rd->domain_id )continue ; -if (active_entry (rd->grant_table, map->ref).frame == mfn ) + act = &_active_entry (rd->grant_table, map->ref); +if ( act->frame == mfn ) (map->flags & GNTMAP_readonly) ? (*rdc)++ : (*wrc)++; }
Yet here I don't understand at all what the change is good for - the only thing you appear to need to change is to insert an underscore.
xen/common/grant_table.c __gnttab_swap_grant_ref(grant_ref_t ref_a, grant_ref_t ref_b) · line 2647
Quoted 13 lines
xen/common/grant_table.c +6
-1@@ -2601,9 +2637,17 @@ __gnttab_swap_grant_ref(grant_ref_t ref_a, grant_ref_t ref_b) {struct domain *d =rcu_lock_current_domain ();struct grant_table *gt = d->grant_table; -struct active_grant_entry *act; +struct active_grant_entry *act_a =NULL ; +struct active_grant_entry *act_b =NULL ;s16 rc = GNTST_okay; +if ( ref_a == ref_b ) +/* +* noop, so avoid acquiring the same active entry +* twice which would case a deadlock.
cause
xen/common/grant_table.c grant_table_create( · line 3004
Quoted 6 lines
xen/common/grant_table.c +1
-1@@ -2953,7 +3001,7 @@ grant_table_create( struct domain *d) {struct grant_table *t; -int i; +int i, j;
unsigned int
xen/common/grant_table.c static void gnttab_usage_print(struct domain *rd) · line 3247
Quoted 6 lines
xen/common/grant_table.c +1
-1@@ -3195,9 +3244,12 @@ static void gnttab_usage_print(struct domain *rd)uint16_t status;uint64_t frame; - act = &active_entry (gt, ref); + act =active_entry_acquire (gt, ref);
As (again in a different context) said recently, debug key handlers would better use try-lock variants, skipping the respective printing if the lock is being held.
And to come back to a remark on v3 and v4: Considering that you drop double_maptrack_lock() here altogether, is patch 1 really correct/sensible in introducing that function in place of double_gt_lock()? Wouldn't the new functionality want the local maptrack and the remote grant table locked in such cases? I.e. isn't one of the locks taken at least pointless?
Jan