Skip to contentSkip to conversation

mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()

PATCH Superseded Qing Wang 3 participants 5 messages May 8 – May 14

Subsystems (2)
lore
Share & tools
lore

Qing Wang

author
#0 lore raw

PATCH 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()

flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache().

There are two paths that call flush_rcu_sheaves_on_cache():

// has cpus_read_lock()
flush_all_rcu_sheaves()

    -> flush_rcu_sheaves_on_cache()

// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()

    -> flush_rcu_sheaves_on_cache()

Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().

Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new l ock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by

- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()

The two orders together would form an AB-BA deadlock.

Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.

---
 mm/slab_common.c | 6 ++++++
 mm/slub.c        | 1 +
 2 files changed, 7 insertions(+)
mm/slab_common.c +6 -0
index d5a70a831a2a..0ee5a4189453 100644
@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
 	if (cache_has_sheaves(s)) {
+		/*
+		 * flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on()
+		 * must be called with the CPU hotplug read lock.
+		 */
+		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
+		cpus_read_unlock();
 		rcu_barrier();
 	}
 
mm/slub.c +1 -0
index 161079ac5ba1..2a005d1e3a74 100644
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
 	struct slub_flush_work *sfw;
 	unsigned int cpu;
 
+	lockdep_assert_cpus_held();
 	mutex_lock(&flush_lock);
 
 	for_each_online_cpu(cpu) {
Signature · 1 line
2.34.1
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 2 lines · Qing Wang

flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online.

Good catch, and also the set of "online cpus" is subject to change if we don't hold the lock.

Quoted 8 lines

But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache().

There are two paths that call flush_rcu_sheaves_on_cache():

// has cpus_read_lock()
flush_all_rcu_sheaves()

    -> flush_rcu_sheaves_on_cache()
Quoted 8 lines

// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()

    -> flush_rcu_sheaves_on_cache()

Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().

Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new l

I would not split the word "lock" to "l" and "ock" and instead start newline before the word :)

Quoted 10 lines

ock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by

- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()

The two orders together would form an AB-BA deadlock.

Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.

Should add:

Fixes 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction")

?

Quoted 11 lines
---
 mm/slab_common.c | 6 ++++++
 mm/slub.c        | 1 +
 2 files changed, 7 insertions(+)
mm/slab_common.c +0 -0
index d5a70a831a2a..0ee5a4189453 100644

mm/slab_common.c EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); · line 2116

Quoted 7 lines
mm/slab_common.c +4 -0
@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
 	if (cache_has_sheaves(s)) {
+		/*
+		 * flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on()
+		 * must be called with the CPU hotplug read lock.
+		 */

nit: not sure this comment is really necessary, given that we can detect it with the lockdep assert added by this patch.

Otherwise LGTM:

Quoted 20 lines
mm/slab_common.c +2 -0
+		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
+		cpus_read_unlock();
 		rcu_barrier();
 	}
 
mm/slub.c +1 -0
index 161079ac5ba1..2a005d1e3a74 100644
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
 	struct slub_flush_work *sfw;
 	unsigned int cpu;
 
+	lockdep_assert_cpus_held();
 	mutex_lock(&flush_lock);
 
 	for_each_online_cpu(cpu) {
Signature · 1 line
2.34.1
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author
#2 lore raw

PATCH v2 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()

flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change.

There are two paths that call flush_rcu_sheaves_on_cache():

// has cpus_read_lock()
flush_all_rcu_sheaves()

    -> flush_rcu_sheaves_on_cache()

// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()

    -> flush_rcu_sheaves_on_cache()

Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().

Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by

- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()

The two orders together would form an AB-BA deadlock.

Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.

Fixes 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction")

---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.

 mm/slab_common.c | 6 ++++++
 mm/slub.c        | 1 +
 2 files changed, 7 insertions(+)
mm/slab_common.c +6 -0
index d5a70a831a2a..0ee5a4189453 100644
@@ -2110,7 +2110,13 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
 	if (cache_has_sheaves(s)) {
+		/*
+		 * flush_rcu_sheaves_on_cache() use queue_work_on() and queue_work_on()
+		 * must be called with the CPU hotplug read lock.
+		 */
+		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
+		cpus_read_unlock();
 		rcu_barrier();
 	}
 
mm/slub.c +1 -0
index 161079ac5ba1..2a005d1e3a74 100644
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
 	struct slub_flush_work *sfw;
 	unsigned int cpu;
 
+	lockdep_assert_cpus_held();
 	mutex_lock(&flush_lock);
 
 	for_each_online_cpu(cpu) {
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v3 1/1 mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()

In reply to Harry Yoo (Oracle)

flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change.

There are two paths that call flush_rcu_sheaves_on_cache():

// has cpus_read_lock()
flush_all_rcu_sheaves()

    -> flush_rcu_sheaves_on_cache()

// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()

    -> flush_rcu_sheaves_on_cache()

Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().

Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by

- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()

The two orders together would form an AB-BA deadlock.

Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.

Fixes 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction")

---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.
Changes in v3:
- Deleted the unnecessary comment.

 mm/slab_common.c | 2 ++
 mm/slub.c        | 1 +
 2 files changed, 3 insertions(+)
mm/slab_common.c +2 -0
index d5a70a831a2a..8b661fff5eed 100644
@@ -2110,7 +2110,9 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
 	if (cache_has_sheaves(s)) {
+		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
+		cpus_read_unlock();
 		rcu_barrier();
 	}
 
mm/slub.c +1 -0
index 161079ac5ba1..2a005d1e3a74 100644
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
 	struct slub_flush_work *sfw;
 	unsigned int cpu;
 
+	lockdep_assert_cpus_held();
 	mutex_lock(&flush_lock);
 
 	for_each_online_cpu(cpu) {
Signature · 1 line
2.34.1
#4 lore raw
Quoted 25 lines · Qing Wang

flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change.

There are two paths that call flush_rcu_sheaves_on_cache():

// has cpus_read_lock()
flush_all_rcu_sheaves()

    -> flush_rcu_sheaves_on_cache()

// no cpus_read_lock()
kvfree_rcu_barrier_on_cache()

    -> flush_rcu_sheaves_on_cache()

Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache().

Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by

- cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...)
- kmem_cache_destroy()

Quoted 7 lines

The two orders together would form an AB-BA deadlock.

Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future.

Fixes 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction")

Added Cc: stable as Harry suggested. Applied to slab/for-next-fixes. Thanks!

Quoted 37 lines

---
Changes in v2:
- Deleted the unnecessary comment.
- Added "Fixes" field in the commit message.
Changes in v3:
- Deleted the unnecessary comment.

 mm/slab_common.c | 2 ++
 mm/slub.c        | 1 +
 2 files changed, 3 insertions(+)
mm/slab_common.c +2 -0
index d5a70a831a2a..8b661fff5eed 100644
@@ -2110,7 +2110,9 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
 	if (cache_has_sheaves(s)) {
+		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
+		cpus_read_unlock();
 		rcu_barrier();
 	}
 
mm/slub.c +1 -0
index 161079ac5ba1..2a005d1e3a74 100644
@@ -4024,6 +4024,7 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
 	struct slub_flush_work *sfw;
 	unsigned int cpu;
 
+	lockdep_assert_cpus_held();
 	mutex_lock(&flush_lock);
 
 	for_each_online_cpu(cpu) {