slab cleanups
Subsystems (2)
v3 cover slab cleanups
Changes since v2:
Adjusted feedback from Vlastimil. Thanks!
Dropped kmalloc subsystem generalization patches. I did more work generalizing kmalloc subsystem. Now they look quite bigger. I'll send them as separate series with RFC.
Changes since v1:
Adjusted comments from Matthew, VLastimil, Rientjes. Thank you for feedback!
Hello, these are cleanup patches for SLUB.
Please consider them for slab-next :)
Any comments will be appreciated.
Thanks.
Hyeonggon Yoo (2):
mm/slub: limit number of node partial slabs only in cache creation mm/slub: refactor deactivate_slab()
mm/slub.c | 105 +++++++++++++++++++++--------------------------------- 1 file changed, 41 insertions(+), 64 deletions(-)
Signature · 1 line
2.33.1
PATCH v3 1/2 mm/slub: limit number of node partial slabs only in cache creation
SLUB sets number of minimum partial slabs for node (min_partial) using set_min_partial(). SLUB holds at least min_partial slabs even if they're empty to avoid excessive use of page allocator.
set_min_partial() limits value of min_partial limits value of min_partial MIN_PARTIAL and MAX_PARTIAL. As set_min_partial() can be called by min_partial_store() too, Only limit value of min_partial in kmem_cache_open() so that it can be changed to value that a user wants.
[ [email protected]: Fold set_min_partial() into its callers ]
--- mm/slub.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-)
mm/slub.c +3 -11
@@ -4000,15 +4000,6 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)return 1 ; } -static void set_min_partial (struct kmem_cache *s,unsigned long min) -{ -if (min <MIN_PARTIAL ) - min =MIN_PARTIAL ; -else if (min >MAX_PARTIAL ) - min =MAX_PARTIAL ; - s->min_partial = min; -} -static void set_cpu_partial (struct kmem_cache *s) {#ifdef CONFIG_SLUB_CPU_PARTIAL
@@ -4215,7 +4206,8 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags)* The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ -set_min_partial (s,ilog2 (s->size) /2 ); + s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); + s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial);set_cpu_partial (s);
@@ -5396,7 +5388,7 @@ static ssize_t min_partial_store(struct kmem_cache *s, const char *buf,if (err)return err; -set_min_partial (s, min); + s->min_partial = min;return length; }SLAB_ATTR (min_partial);
Signature · 1 line
2.33.1
Re: [PATCH v3 1/2] mm/slub: limit number of node partial slabs only in cache creation
Quoted 9 lines · Hyeonggon Yoo
SLUB sets number of minimum partial slabs for node (
min_partial) usingset_min_partial(). SLUB holds at leastmin_partialslabs even if they're empty to avoid excessive use of page allocator.
set_min_partial()limits value ofmin_partiallimits value ofmin_partialMIN_PARTIALandMAX_PARTIAL. Asset_min_partial()can be called bymin_partial_store()too, Only limit value ofmin_partialinkmem_cache_open()so that it can be changed to value that a user wants.
Quoted 4 lines
[ [email protected]: Fold
set_min_partial()into its callers ]Signed-off-by Hyeonggon Yoo <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>
LGTM!
PATCH v3 2/2 mm/slub: refactor deactivate_slab()
In reply to Hyeonggon YooSimplify deactivate_slab() by unlocking n->list_lock and retrying cmpxchg_double() when cmpxchg_double() fails, and perform add_{partial,full} only when it succeed.
Releasing and taking n->list_lock again here is not harmful as SLUB avoids deactivating slabs as much as possible.
[ [email protected]: perform add_{partial,full} when cmpxchg_double()
succeed.
count deactivating full slabs even if debugging flag is not set. ]
--- mm/slub.c | 91 +++++++++++++++++++++++-------------------------------- 1 file changed, 38 insertions(+), 53 deletions(-)
mm/slub.c +38 -53
@@ -2348,10 +2348,10 @@ static void init_kmem_cache_cpus(struct kmem_cache *s)static void deactivate_slab (struct kmem_cache *s,struct slab *slab,void *freelist) { -enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE }; +enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE ,M_FULL_NOLIST };struct kmem_cache_node *n =get_node (s,slab_nid (slab)); -int lock =0 , free_delta =0 ; -enum slab_modes l =M_NONE , m =M_NONE ; +int free_delta =0 ; +enum slab_modes mode =M_NONE ;void *nextfree, *freelist_iter, *freelist_tail;int tail =DEACTIVATE_TO_HEAD ;unsigned long flags =0 ;
@@ -2393,14 +2393,10 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab,* Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * -* We setup the list membership and then perform a cmpxchg -* with the count. If there is a mismatch then the slab -* is not unfrozen but the slab is on the wrong list. -* -* Then we restart the process which may have to remove -* the slab from the list that we just put it on again -* because the number of objects in the slab may have -* changed. +* We first perform cmpxchg holding lock and insert to list +* when it succeed. If there is mismatch then the slab is not +* unfrozen and number of objects in the slab may have changed. +* Then release lock and retry cmpxchg again. */ redo:
@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) { - m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full +* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ; -if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL ) -add_full (s, n, slab); - } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags); goto redo; + } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - } + }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags); +stat (s,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL ); }#ifdef CONFIG_SLUB_CPU_PARTIAL
Signature · 1 line
2.33.1
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
Quoted 7 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.
Quoted 6 lines
[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>
adding both to slab-next. Fixed up some nits myself, see below:
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2420
Quoted 8 lines
mm/slub.c +1
-1@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) {
This was against kernel style even before the patch - we use { } in the 'else if' branch, thus all branches should use { } even if one-line.
Quoted 33 lines
mm/slub.c +6
-27- m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags);
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2436
Quoted 10 lines
mm/slub.c +10
-0+ }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full +* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ;
Ditto here (this is new).
Quoted 29 lines
mm/slub.c +8
-13-if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL ) -add_full (s, n, slab); - } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags); goto redo; + } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2462
Quoted 10 lines
mm/slub.c +6
-1stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - } + }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags); +stat (s,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL );
And here.
Quoted 3 lines
mm/slub.c +0
-0}#ifdef CONFIG_SLUB_CPU_PARTIAL
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
Quoted 23 lines · Vlastimil Babka
Quoted 7 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.Quoted 6 lines
[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>adding both to slab-next. Fixed up some nits myself, see below:
Quoted 4 lines
mm/slub.c +0
-0@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;
Quoted 8 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2420
Quoted 4 lines
mm/slub.c +1
-1if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) {This was against kernel style even before the patch - we use { } in the 'else if' branch, thus all branches should use { } even if one-line.
Ah, you are right. Agree with this change. "Remove unnecessary brace" rule does not apply here.
Quoted 37 lines
Quoted 37 lines
mm/slub.c +10
-27- m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full
Quoted 8 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2436
Quoted 6 lines
mm/slub.c +6
-0+* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ;Ditto here (this is new).
Yes.
Quoted 34 lines
Quoted 34 lines
mm/slub.c +8
-14-if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL ) -add_full (s, n, slab); - } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags); goto redo; + } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - }
Quoted 9 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2462
Quoted 6 lines
mm/slub.c +6
-0+ }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags); +stat (s,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL );And here.
Yes.
Quoted 4 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2465
Quoted 3 lines
mm/slub.c +0
-0}#ifdef CONFIG_SLUB_CPU_PARTIAL
Thanks!
Signature · 2 lines
Thank you, You are awesome! Hyeonggon :-)
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
In reply to Hyeonggon YooHello,
Quoted 110 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 91 +++++++++++++++++++++++-------------------------------- 1 file changed, 38 insertions(+), 53 deletions(-)mm/slub.c +24
-43@@ -2348,10 +2348,10 @@ static void init_kmem_cache_cpus(struct kmem_cache *s)static void deactivate_slab (struct kmem_cache *s,struct slab *slab,void *freelist) { -enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE }; +enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE ,M_FULL_NOLIST };struct kmem_cache_node *n =get_node (s,slab_nid (slab)); -int lock =0 , free_delta =0 ; -enum slab_modes l =M_NONE , m =M_NONE ; +int free_delta =0 ; +enum slab_modes mode =M_NONE ;void *nextfree, *freelist_iter, *freelist_tail;int tail =DEACTIVATE_TO_HEAD ;unsigned long flags =0 ;@@ -2393,14 +2393,10 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab,* Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * -* We setup the list membership and then perform a cmpxchg -* with the count. If there is a mismatch then the slab -* is not unfrozen but the slab is on the wrong list. -* -* Then we restart the process which may have to remove -* the slab from the list that we just put it on again -* because the number of objects in the slab may have -* changed. +* We first perform cmpxchg holding lock and insert to list +* when it succeed. If there is mismatch then the slab is not +* unfrozen and number of objects in the slab may have changed. +* Then release lock and retry cmpxchg again. */ redo:@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) { - m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full +* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ; -if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL ) -add_full (s, n, slab);
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2444
Quoted 10 lines
mm/slub.c +3
-3- } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags);
The slab doesn't belong to any node here, should we remove locking/unlocking spin for cmpxchg_double_slab() call? Just calling spin_lock_irqsave() before add_partial()/add_full call is fine?
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2445
Quoted 1 line
mm/slub.c +0
-0goto redo;
How about do {...} while(!cmpxchg_double_slab())? The readability looks better?
Regards,
Xiongwei
Quoted 31 lines
mm/slub.c +11
-8+ } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - } + }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags); +stat (s,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL ); }#ifdef CONFIG_SLUB_CPU_PARTIAL --2.33.1
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
Quoted 109 lines · Xiongwei Song
Hello,
Quoted 106 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 91 +++++++++++++++++++++++-------------------------------- 1 file changed, 38 insertions(+), 53 deletions(-)mm/slub.c +24
-39@@ -2348,10 +2348,10 @@ static void init_kmem_cache_cpus(struct kmem_cache *s)static void deactivate_slab (struct kmem_cache *s,struct slab *slab,void *freelist) { -enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE }; +enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE ,M_FULL_NOLIST };struct kmem_cache_node *n =get_node (s,slab_nid (slab)); -int lock =0 , free_delta =0 ; -enum slab_modes l =M_NONE , m =M_NONE ; +int free_delta =0 ; +enum slab_modes mode =M_NONE ;void *nextfree, *freelist_iter, *freelist_tail;int tail =DEACTIVATE_TO_HEAD ;unsigned long flags =0 ;@@ -2393,14 +2393,10 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab,* Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * -* We setup the list membership and then perform a cmpxchg -* with the count. If there is a mismatch then the slab -* is not unfrozen but the slab is on the wrong list. -* -* Then we restart the process which may have to remove -* the slab from the list that we just put it on again -* because the number of objects in the slab may have -* changed. +* We first perform cmpxchg holding lock and insert to list +* when it succeed. If there is mismatch then the slab is not +* unfrozen and number of objects in the slab may have changed. +* Then release lock and retry cmpxchg again. */ redo:@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) { - m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full +* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ;
Quoted 3 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2455
Quoted 3 lines
mm/slub.c +0
-3-if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL )
i> > - add_full(s, n, slab);
Quoted 11 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2444
Quoted 10 lines
mm/slub.c +3
-3- } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags);
Quoted 4 lines
The slab doesn't belong to any node here, should we remove locking/unlocking spin for
cmpxchg_double_slab()call? Just callingspin_lock_irqsave()before add_partial()/add_full call is fine?
I thought about that, and tested, but that is not okay.
taking spinlock around cmpxchg prevents race between __slab_free() and deactivate_slab(). list can be corrupted without spinlock.
think about case below: (when SLAB_STORE_USER is set)
__slab_free() deactivate_slab()
================= =================
(deactivating full slab) cmpxchg_double()
spin_lock_irqsave()cmpxchg_double()
/* not in full list yet */remove_full()add_partial()spin_unlock_irqrestore()
spin_lock_irqsave() add_full() spin_unlock_irqrestore()
Quoted 4 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2445
Quoted 1 line
mm/slub.c +0
-0goto redo;How about do {...} while(!cmpxchg_double_slab())? The readability looks better?
This will be:
do {
if (mode == M_PARTIAL || mode == M_FULL)
spin_unlock_irqrestore();
[...]} while (!cmpxchg_double_slab());
I think goto version is better for reading?
Thanks!
Quoted 34 lines
Regards,
XiongweiQuoted 31 lines
mm/slub.c +11
-8+ } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - } + }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags);y +stat (s ,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL ); }#ifdef CONFIG_SLUB_CPU_PARTIAL --2.33.1
Signature · 2 lines
Thank you, You are awesome! Hyeonggon :-)
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
Quoted 148 lines · Hyeonggon Yoo
Quoted 109 lines · Xiongwei Song
Hello,
Quoted 106 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 91 +++++++++++++++++++++++-------------------------------- 1 file changed, 38 insertions(+), 53 deletions(-)mm/slub.c +24
-39@@ -2348,10 +2348,10 @@ static void init_kmem_cache_cpus(struct kmem_cache *s)static void deactivate_slab (struct kmem_cache *s,struct slab *slab,void *freelist) { -enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE }; +enum slab_modes {M_NONE ,M_PARTIAL ,M_FULL ,M_FREE ,M_FULL_NOLIST };struct kmem_cache_node *n =get_node (s,slab_nid (slab)); -int lock =0 , free_delta =0 ; -enum slab_modes l =M_NONE , m =M_NONE ; +int free_delta =0 ; +enum slab_modes mode =M_NONE ;void *nextfree, *freelist_iter, *freelist_tail;int tail =DEACTIVATE_TO_HEAD ;unsigned long flags =0 ;@@ -2393,14 +2393,10 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab,* Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * -* We setup the list membership and then perform a cmpxchg -* with the count. If there is a mismatch then the slab -* is not unfrozen but the slab is on the wrong list. -* -* Then we restart the process which may have to remove -* the slab from the list that we just put it on again -* because the number of objects in the slab may have -* changed. +* We first perform cmpxchg holding lock and insert to list +* when it succeed. If there is mismatch then the slab is not +* unfrozen and number of objects in the slab may have changed. +* Then release lock and retry cmpxchg again. */ redo:@@ -2420,61 +2416,50 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen =0 ;if (!new.inuse && n->nr_partial >= s->min_partial) - m =M_FREE ; + mode =M_FREE ;else if (new.freelist) { - m =M_PARTIAL ; -if (!lock) { - lock =1 ; -/* -* Taking the spinlock removes the possibility that -* acquire_slab() will see a slab that is frozen -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - }else { - m =M_FULL ; -if (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && !lock) { - lock =1 ; -/* -* This also ensures that the scanning of full -* slabs from diagnostic functions will not see -* any frozen slabs. -*/ -spin_lock_irqsave (&n->list_lock, flags); - } - } - -if (l != m) { -if (l ==M_PARTIAL ) -remove_partial (n, slab); -else if (l ==M_FULL ) -remove_full (s, n, slab); + mode =M_PARTIAL ; +/* +* Taking the spinlock removes the possibility that +* acquire_slab() will see a slab that is frozen +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else if (kmem_cache_debug_flags (s,SLAB_STORE_USER )) { + mode =M_FULL ; +/* +* This also ensures that the scanning of full +* slabs from diagnostic functions will not see +* any frozen slabs. +*/ +spin_lock_irqsave (&n->list_lock, flags); + }else + mode =M_FULL_NOLIST ;Quoted 3 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2455
Quoted 3 lines
mm/slub.c +0
-3-if (m ==M_PARTIAL ) -add_partial (n, slab, tail); -else if (m ==M_FULL )i> > -
add_full(s, n, slab);Quoted 11 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2444
Quoted 10 lines
mm/slub.c +3
-3- } - l = m;if (!cmpxchg_double_slab (s, slab, old.freelist, old.counters, new.freelist, new.counters, -"unfreezing slab" )) +"unfreezing slab" )) { +if (mode ==M_PARTIAL || mode ==M_FULL ) +spin_unlock_irqrestore (&n->list_lock, flags);Quoted 4 lines
The slab doesn't belong to any node here, should we remove locking/unlocking spin for
cmpxchg_double_slab()call? Just callingspin_lock_irqsave()before add_partial()/add_full call is fine?I thought about that, and tested, but that is not okay.
taking spinlock around cmpxchg prevents race between
__slab_free()anddeactivate_slab(). list can be corrupted without spinlock.think about case below: (when
SLAB_STORE_USERis set)
__slab_free()deactivate_slab()
================= =================
(deactivating full slab)cmpxchg_double()
spin_lock_irqsave()cmpxchg_double()
Quoted 8 lines
/* not in full list yet */
remove_full()add_partial()spin_unlock_irqrestore()spin_lock_irqsave() add_full() spin_unlock_irqrestore()
Oh... Looks reasonable. Thanks for the detailed explanation.
Quoted 9 lines
Quoted 6 lines
mm/slub.c static void deactivate_slab(struct kmem_cache *s, struct slab *slab, · line 2445
Quoted 3 lines
mm/slub.c +0
-0goto redo;How about do {...} while(!cmpxchg_double_slab())? The readability looks better?
This will be:
Quoted 8 lines
do { if (mode == M_PARTIAL || mode == M_FULL) spin_unlock_irqrestore(); [...]} while (!cmpxchg_double_slab());
I saw __slab_free() is doing so. Not a big deal.
Regards,
Xiongwei
Quoted 42 lines
I think goto version is better for reading?
Thanks!
Quoted 34 lines
Regards,
XiongweiQuoted 31 lines
mm/slub.c +11
-8+ } -if (lock) -spin_unlock_irqrestore (&n->list_lock, flags); -if (m ==M_PARTIAL ) +if (mode ==M_PARTIAL ) { +add_partial (n, slab, tail); +spin_unlock_irqrestore (&n->list_lock, flags);stat (s, tail); -else if (m ==M_FULL ) -stat (s,DEACTIVATE_FULL ); -else if (m ==M_FREE ) { + }else if (mode ==M_FREE ) {stat (s,DEACTIVATE_EMPTY );discard_slab (s, slab);stat (s,FREE_SLAB ); - } + }else if (mode ==M_FULL ) { +add_full (s, n, slab); +spin_unlock_irqrestore (&n->list_lock, flags);y +stat (s ,DEACTIVATE_FULL ); + }else if (mode ==M_FULL_NOLIST ) +stat (s,DEACTIVATE_FULL ); }#ifdef CONFIG_SLUB_CPU_PARTIAL --2.33.1
--
Thank you, You are awesome!
Hyeonggon :-)
Re: [PATCH v3 2/2] mm/slub: refactor deactivate_slab()
In reply to Hyeonggon YooQuoted 7 lines · Hyeonggon Yoo
Simplify
deactivate_slab()by unlockingn->list_lockand retryingcmpxchg_double()whencmpxchg_double()fails, and perform add_{partial,full} only when it succeed.Releasing and taking
n->list_lockagain here is not harmful as SLUB avoids deactivating slabs as much as possible.
Quoted 6 lines
[ [email protected]: perform add_{partial,full} when
cmpxchg_double()
succeed.count deactivating full slabs even if debugging flag is not set. ]
Signed-off-by Hyeonggon Yoo <[email protected]>
Thanks!