mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()
Subsystems (2)
PATCH 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()
flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache().
There are two paths that call flush_rcu_sheaves_on_cache():
// has cpus_read_lock()
flush_all_rcu_sheaves()
-> flush_rcu_sheaves_on_cache()
// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()
-> flush_rcu_sheaves_on_cache()
Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().
Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new l ock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by
- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()
The two orders together would form an AB-BA deadlock.
Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.
--- mm/slab_common.c | 6 ++++++ mm/slub.c | 1 + 2 files changed, 7 insertions(+)
mm/slab_common.c +6 -0
@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) {if (cache_has_sheaves (s)) { +/* +* flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on() +* must be called with the CPU hotplug read lock. +*/ +cpus_read_lock ();flush_rcu_sheaves_on_cache (s); +cpus_read_unlock ();rcu_barrier (); }
mm/slub.c +1 -0
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)struct slub_flush_work *sfw;unsigned int cpu; +lockdep_assert_cpus_held ();mutex_lock (&flush_lock);for_each_online_cpu (cpu) {
Signature · 1 line
2.34.1
Quoted 2 lines · Qing Wang
flush_rcu_sheaves_on_cache()callsqueue_work_on()in afor_each_online_cpu()loop, which requires the cpu to stay online.
Good catch, and also the set of "online cpus" is subject to change if we don't hold the lock.
Quoted 8 lines
But
cpus_read_lock()is not held inkvfree_rcu_barrier_on_cache().There are two paths that call
flush_rcu_sheaves_on_cache():// has
cpus_read_lock()
flush_all_rcu_sheaves()-> flush_rcu_sheaves_on_cache()
Quoted 8 lines
// no
cpus_read_lock()
kvfree_rcu_barrier_on_cache()-> flush_rcu_sheaves_on_cache()Fix this by holding
cpus_read_lock()inkvfree_rcu_barrier_on_cache().Why not move
cpus_read_lock()fromflush_all_rcu_sheaves()intoflush_rcu_sheaves_on_cache()? The reason is it would introduce a new l
I would not split the word "lock" to "l" and "ock" and instead start newline before the word :)
Quoted 10 lines
ock order (
slab_mutex->cpu_hotplug_lock). The reverse order (cpu_hotplug_lock->slab_mutex) is established by-
cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
-kmem_cache_destroy()The two orders together would form an AB-BA deadlock.
Finally, add
lockdep_assert_cpus_held()inflush_rcu_sheaves_on_cache()to catch the same problem in the future.
Should add:
?
Quoted 11 lines
Signed-off-by Qing Wang <[email protected]>--- mm/slab_common.c | 6 ++++++ mm/slub.c | 1 + 2 files changed, 7 insertions(+)mm/slab_common.c +0
-0
mm/slab_common.c EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); · line 2116
Quoted 7 lines
mm/slab_common.c +4
-0@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) {if (cache_has_sheaves (s)) { +/* +* flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on() +* must be called with the CPU hotplug read lock. +*/
nit: not sure this comment is really necessary, given that we can detect it with the lockdep assert added by this patch.
Otherwise LGTM:
Quoted 20 lines
mm/slab_common.c +2
-0+cpus_read_lock ();flush_rcu_sheaves_on_cache (s); +cpus_read_unlock ();rcu_barrier (); }mm/slub.c +1
-0@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)struct slub_flush_work *sfw;unsigned int cpu; +lockdep_assert_cpus_held ();mutex_lock (&flush_lock);for_each_online_cpu (cpu) {Signature · 1 line
2.34.1
Signature · 2 lines
Cheers, Harry / Hyeonggon
PATCH v2 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()
flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change.
There are two paths that call flush_rcu_sheaves_on_cache():
// has cpus_read_lock()
flush_all_rcu_sheaves()
-> flush_rcu_sheaves_on_cache()
// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()
-> flush_rcu_sheaves_on_cache()
Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().
Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by
- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()
The two orders together would form an AB-BA deadlock.
Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.
---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.
mm/slab_common.c | 6 ++++++ mm/slub.c | 1 + 2 files changed, 7 insertions(+)
mm/slab_common.c +6 -0
@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) {if (cache_has_sheaves (s)) { +/* +* flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on() +* must be called with the CPU hotplug read lock. +*/ +cpus_read_lock ();flush_rcu_sheaves_on_cache (s); +cpus_read_unlock ();rcu_barrier (); }
mm/slub.c +1 -0
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)struct slub_flush_work *sfw;unsigned int cpu; +lockdep_assert_cpus_held ();mutex_lock (&flush_lock);for_each_online_cpu (cpu) {
Signature · 1 line
2.34.1
PATCH v3 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()
In reply to Harry Yoo (Oracle)flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change.
There are two paths that call flush_rcu_sheaves_on_cache():
// has cpus_read_lock()
flush_all_rcu_sheaves()
-> flush_rcu_sheaves_on_cache()
// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()
-> flush_rcu_sheaves_on_cache()
Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().
Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by
- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()
The two orders together would form an AB-BA deadlock.
Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.
---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.
Changes in v3:
- Deleted the unnecessary comment.
mm/slab_common.c | 2 ++ mm/slub.c | 1 + 2 files changed, 3 insertions(+)
mm/slab_common.c +2 -0
@@ -2110,7 +2110,9 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) {if (cache_has_sheaves (s)) { +cpus_read_lock ();flush_rcu_sheaves_on_cache (s); +cpus_read_unlock ();rcu_barrier (); }
mm/slub.c +1 -0
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)struct slub_flush_work *sfw;unsigned int cpu; +lockdep_assert_cpus_held ();mutex_lock (&flush_lock);for_each_online_cpu (cpu) {
Signature · 1 line
2.34.1
Vlastimil Babka (SUSE)
maintainerQuoted 25 lines · Qing Wang
flush_rcu_sheaves_on_cache()callsqueue_work_on()in afor_each_online_cpu()loop, which requires the cpu to stay online. Butcpus_read_lock()is not held inkvfree_rcu_barrier_on_cache()and the set of "online cpus" is subject to change.There are two paths that call
flush_rcu_sheaves_on_cache():// has
cpus_read_lock()
flush_all_rcu_sheaves()-> flush_rcu_sheaves_on_cache()// no
cpus_read_lock()
kvfree_rcu_barrier_on_cache()-> flush_rcu_sheaves_on_cache()Fix this by holding
cpus_read_lock()inkvfree_rcu_barrier_on_cache().Why not move
cpus_read_lock()fromflush_all_rcu_sheaves()intoflush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex->cpu_hotplug_lock). The reverse order (cpu_hotplug_lock->slab_mutex) is established by-
cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
-kmem_cache_destroy()
Quoted 7 lines
The two orders together would form an AB-BA deadlock.
Finally, add
lockdep_assert_cpus_held()inflush_rcu_sheaves_on_cache()to catch the same problem in the future.Fixes 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction")Signed-off-by Qing Wang <[email protected]>
Added Cc: stable as Harry suggested. Applied to slab/for-next-fixes. Thanks!
Quoted 37 lines
---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.
Changes in v3:
- Deleted the unnecessary comment.mm/slab_common.c | 2 ++ mm/slub.c | 1 + 2 files changed, 3 insertions(+)mm/slab_common.c +2
-0@@ -2110,7 +2110,9 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) {if (cache_has_sheaves (s)) { +cpus_read_lock ();flush_rcu_sheaves_on_cache (s); +cpus_read_unlock ();rcu_barrier (); }mm/slub.c +1
-0@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)struct slub_flush_work *sfw;unsigned int cpu; +lockdep_assert_cpus_held ();mutex_lock (&flush_lock);for_each_online_cpu (cpu) {