Skip to contentSkip to conversation

tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem

PATCH RESEND In review Joel Fernandes (Google) 3 participants 10 messages Jan 24, 2023 – Jan 28, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH RESEND v2 1/1 tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")

---
Sorry, resending with CC to stable.

 drivers/base/cpu.c       |  3 ++-
 include/linux/tick.h     |  2 ++
 kernel/time/tick-sched.c | 11 ++++++++---
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
 	struct device *dev = get_cpu_device(cpu);
-	return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+	return dev && container_of(dev, struct cpu, dev)->hotpluggable
+		&& tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
 				     enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
 				       enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -3
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
 	tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
 	/*
 	 * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
 	 * CPUs. It must remain online when nohz full is enabled.
 	 */
 	if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-		return -EBUSY;
-	return 0;
+		return false;
+	return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+	return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
Signature · 1 line
2.39.1.405.gd4c25cc71f-goog
maintainer
#1 lore raw
Quoted 13 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Quoted 9 lines
Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

 drivers/base/cpu.c       |  3 ++-
 include/linux/tick.h     |  2 ++
 kernel/time/tick-sched.c | 11 ++++++++---
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
 	struct device *dev = get_cpu_device(cpu);
-	return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+	return dev && container_of(dev, struct cpu, dev)->hotpluggable
+		&& tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
 				     enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
 				       enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -3
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
 	tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
 	/*
 	 * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
 	 * CPUs. It must remain online when nohz full is enabled.
 	 */
 	if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-		return -EBUSY;
-	return 0;
+		return false;
+	return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+	return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
Signature · 1 line
2.39.1.405.gd4c25cc71f-goog
#2 lore raw
Quoted 25 lines · Paul E. McKenney
Quoted 13 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Quoted 9 lines
Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

[1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/
[2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/
[3] http://140.211.169.189/linux-stable-rc/

Happy to benefit the community and this is a fruitful learning journey ;-)

Cheers
Zhouyi

Quoted 77 lines

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

 drivers/base/cpu.c       |  3 ++-
 include/linux/tick.h     |  2 ++
 kernel/time/tick-sched.c | 11 ++++++++---
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
      struct device *dev = get_cpu_device(cpu);
-     return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+     return dev && container_of(dev, struct cpu, dev)->hotpluggable
+             && tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
                                   enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
                                     enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -4
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
      tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
      /*
       * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
       * CPUs. It must remain online when nohz full is enabled.
       */
      if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-             return -EBUSY;
-     return 0;
+             return false;
+     return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+     return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
--

2.39.1.405.gd4c25cc71f-goog

author Tested-by Zhouyi Zhou
#3 lore raw
Quoted 39 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 15 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Quoted 9 lines
Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

Quoted 5 lines

[1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/
[2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/
[3] http://140.211.169.189/linux-stable-rc/

Happy to benefit the community and this is a fruitful learning journey ;-)

Thanks a lot Zhouyi! After testing something your reply can also optionally include:

Thanks.

Have fun at the university. I miss my university days.

- Joel

Quoted 80 lines

Cheers
Zhouyi

Quoted 77 lines

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

drivers/base/cpu.c       |  3 ++-
include/linux/tick.h     |  2 ++
kernel/time/tick-sched.c | 11 ++++++++---
3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
     struct device *dev = get_cpu_device(cpu);
-     return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+     return dev && container_of(dev, struct cpu, dev)->hotpluggable
+             && tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
                                  enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
                                    enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
 * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -4
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
     tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
     /*
      * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
      * CPUs. It must remain online when nohz full is enabled.
      */
     if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-             return -EBUSY;
-     return 0;
+             return false;
+     return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+     return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
--

2.39.1.405.gd4c25cc71f-goog

#4 lore raw
Quoted 44 lines · Joel Fernandes
Quoted 44 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 15 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Quoted 9 lines
Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

[1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/
[2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/
[3] http://140.211.169.189/linux-stable-rc/

Quoted 7 lines
Quoted 1 line

Happy to benefit the community and this is a fruitful learning journey ;-)

Thanks a lot Zhouyi! After testing something your reply can also optionally include:

Thanks.

I see. Thank Joel for your guidance and your encouragement ;-)

Quoted 2 lines

Have fun at the university. I miss my university days.

Many thanks, I also miss my university days and days as a graduate student [1], but I am no longer young ;-)
The open source Lab of Oregon State University lent me their VM hoping to contribute to the open source
community, but I feel like being in university again while I am having fun in the VM ;-)

[1] https://wiki.freebsd.org/ZhouyiZHOU

Have a nice day ;-)
Cheers
Zhouyi

Quoted 84 lines

- Joel

Quoted 80 lines

Cheers
Zhouyi

Quoted 77 lines

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

drivers/base/cpu.c       |  3 ++-
include/linux/tick.h     |  2 ++
kernel/time/tick-sched.c | 11 ++++++++---
3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
     struct device *dev = get_cpu_device(cpu);
-     return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+     return dev && container_of(dev, struct cpu, dev)->hotpluggable
+             && tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
                                  enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
                                    enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
 * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -4
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
     tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
     /*
      * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
      * CPUs. It must remain online when nohz full is enabled.
      */
     if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-             return -EBUSY;
-     return 0;
+             return false;
+     return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+     return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
--

2.39.1.405.gd4c25cc71f-goog

#5 lore raw
In reply to Zhouyi Zhou
Quoted 27 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 22 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 8 lines
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-(

[    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
0x00000008
...
[    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
scsi_end_request+0xd8/0x1f0 (unreliable)^M
[    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
scsi_io_completion+0x88/0x700^M
[    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
scsi_finish_command+0xe8/0x150^M
[    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
scsi_complete+0x90/0x140^M
[    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
blk_complete_reqs+0x80/0xa0^M
[    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
__do_softirq+0x1e0/0x4e0^M
[    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
do_softirq_own_stack+0x48/0x60^M
[    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
0xc00000000a5e7c80^M
[    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
do_softirq+0xb0/0xc0^M
[    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
__local_bh_enable_ip+0xdc/0x110^M
[    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
irq_forced_thread_fn+0xc8/0xf0^M
[    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
irq_thread+0x1b4/0x2a0^M
[    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
kthread+0x1a8/0x1d0^M
[    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]

ret_from_kernel_thread+0x5c/0x64^M

But when I invoked [5] by hand, the bug did not show again [6].

[4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
[5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
[6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log

I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it.

Sorry for the inconvenience that I bring

Thanks
Zhouyi

Quoted 86 lines

[1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/
[2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/
[3] http://140.211.169.189/linux-stable-rc/

Happy to benefit the community and this is a fruitful learning journey ;-)

Cheers
Zhouyi

Quoted 77 lines

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

 drivers/base/cpu.c       |  3 ++-
 include/linux/tick.h     |  2 ++
 kernel/time/tick-sched.c | 11 ++++++++---
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
      struct device *dev = get_cpu_device(cpu);
-     return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+     return dev && container_of(dev, struct cpu, dev)->hotpluggable
+             && tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
                                   enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
                                     enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -4
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
      tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
      /*
       * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
       * CPUs. It must remain online when nohz full is enabled.
       */
      if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-             return -EBUSY;
-     return 0;
+             return false;
+     return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+     return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
--

2.39.1.405.gd4c25cc71f-goog

Tested-by
#6 lore raw
Quoted 78 lines · Zhouyi Zhou
Quoted 27 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 22 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 8 lines
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-(

[    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
0x00000008
...
[    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
scsi_end_request+0xd8/0x1f0 (unreliable)^M
[    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
scsi_io_completion+0x88/0x700^M
[    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
scsi_finish_command+0xe8/0x150^M
[    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
scsi_complete+0x90/0x140^M
[    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
blk_complete_reqs+0x80/0xa0^M
[    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
__do_softirq+0x1e0/0x4e0^M
[    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
do_softirq_own_stack+0x48/0x60^M
[    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
0xc00000000a5e7c80^M
[    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
do_softirq+0xb0/0xc0^M
[    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
__local_bh_enable_ip+0xdc/0x110^M
[    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
irq_forced_thread_fn+0xc8/0xf0^M
[    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
irq_thread+0x1b4/0x2a0^M
[    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
kthread+0x1a8/0x1d0^M
[    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]

ret_from_kernel_thread+0x5c/0x64^M

But when I invoked [5] by hand, the bug did not show again [6].

[4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
[5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
[6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log

Quoted 8 lines

I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it.

Sorry for the inconvenience that I bring

Thanks
Zhouyi

Hi the above kernel null pointer dereference bug has nothing to do with Joel's fix because I can reproduce it on original 5.15.y [7] using as while loop [8] (after 36 iterations, the bug fires). So, Joel's patch is tested good!

In following days, I am like to dig out the root of the null pointer dereference bug.
[7] http://140.211.169.189/stable-for-joel/linux-stable-rc/
[8] http://140.211.169.189/stable-for-joel/linux-stable-rc/qemu-cmd.sh

Thanks
Zhouyi

Quoted 86 lines
Quoted 86 lines

[1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/
[2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/
[3] http://140.211.169.189/linux-stable-rc/

Happy to benefit the community and this is a fruitful learning journey ;-)

Cheers
Zhouyi

Quoted 77 lines

Thanx, Paul

Quoted 74 lines

---
Sorry, resending with CC to stable.

 drivers/base/cpu.c       |  3 ++-
 include/linux/tick.h     |  2 ++
 kernel/time/tick-sched.c | 11 ++++++++---
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/base/cpu.c +2 -1
index 55405ebf23ab..450dca235a2f 100644
@@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = {
 bool cpu_is_hotpluggable(unsigned int cpu)
 {
      struct device *dev = get_cpu_device(cpu);
-     return dev && container_of(dev, struct cpu, dev)->hotpluggable;
+     return dev && container_of(dev, struct cpu, dev)->hotpluggable
+             && tick_nohz_cpu_hotpluggable(cpu);
 }
 EXPORT_SYMBOL_GPL(cpu_is_hotpluggable);
 
include/linux/tick.h +2 -0
index bfd571f18cfd..9459fef5b857 100644
@@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
                                   enum tick_dep_bits bit);
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
                                     enum tick_dep_bits bit);
+extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
@@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
 static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
 static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; }
 
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +8 -4
index 9c6f661fb436..63e3e8ebcd64 100644
@@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
      tick_nohz_full_running = true;
 }
 
-static int tick_nohz_cpu_down(unsigned int cpu)
+bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
      /*
       * The tick_do_timer_cpu CPU handles housekeeping duty (unbound
@@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu)
       * CPUs. It must remain online when nohz full is enabled.
       */
      if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
-             return -EBUSY;
-     return 0;
+             return false;
+     return true;
+}
+
+static int tick_nohz_cpu_down(unsigned int cpu)
+{
+     return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY;
 }
 
 void __init tick_nohz_init(void)
--

2.39.1.405.gd4c25cc71f-goog

#7 lore raw
Quoted 86 lines · Zhouyi Zhou
Quoted 84 lines · Zhouyi Zhou
Quoted 27 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 22 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 8 lines
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-(

[    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
0x00000008
...
[    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
scsi_end_request+0xd8/0x1f0 (unreliable)^M
[    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
scsi_io_completion+0x88/0x700^M
[    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
scsi_finish_command+0xe8/0x150^M
[    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
scsi_complete+0x90/0x140^M
[    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
blk_complete_reqs+0x80/0xa0^M
[    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
__do_softirq+0x1e0/0x4e0^M
[    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
do_softirq_own_stack+0x48/0x60^M
[    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
0xc00000000a5e7c80^M
[    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
do_softirq+0xb0/0xc0^M
[    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
__local_bh_enable_ip+0xdc/0x110^M
[    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
irq_forced_thread_fn+0xc8/0xf0^M
[    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
irq_thread+0x1b4/0x2a0^M
[    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
kthread+0x1a8/0x1d0^M
[    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]

ret_from_kernel_thread+0x5c/0x64^M

But when I invoked [5] by hand, the bug did not show again [6].

[4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
[5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
[6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log

I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it.

Sorry for the inconvenience that I bring

Quoted 7 lines
Quoted 2 lines

Thanks
Zhouyi

Hi the above kernel null pointer dereference bug has nothing to do with Joel's fix because I can reproduce it on original 5.15.y [7] using as while loop [8] (after 36 iterations, the bug fires). So, Joel's patch is tested good!

Interesting. I have been running rcutorture's TREE03 on 5.15.y quite a lot and I don't see such an issue.

However, your logs showed the crash is SCSI related. These were the
recent SCSI commits in 5.15.y but I am not sure if it causes the issue:

13259b6 scsi: mpi3mr: Refer CONFIG_SCSI_MPI3MR in Makefile 513fdf0 scsi: ufs: Stop using the clock scaling lock in the error handler 7c26d21 scsi: ufs: core: WLUN suspend SSU/enter hibern

Perhaps report it to the scsi and/or stable lists, or do some web searches for if someone else sees it.

- Joel

#8 lore raw
Quoted 103 lines · Joel Fernandes
Quoted 86 lines · Zhouyi Zhou
Quoted 84 lines · Zhouyi Zhou
Quoted 27 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 22 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 8 lines
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-(

[    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
0x00000008
...
[    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
scsi_end_request+0xd8/0x1f0 (unreliable)^M
[    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
scsi_io_completion+0x88/0x700^M
[    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
scsi_finish_command+0xe8/0x150^M
[    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
scsi_complete+0x90/0x140^M
[    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
blk_complete_reqs+0x80/0xa0^M
[    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
__do_softirq+0x1e0/0x4e0^M
[    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
do_softirq_own_stack+0x48/0x60^M
[    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
0xc00000000a5e7c80^M
[    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
do_softirq+0xb0/0xc0^M
[    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
__local_bh_enable_ip+0xdc/0x110^M
[    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
irq_forced_thread_fn+0xc8/0xf0^M
[    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
irq_thread+0x1b4/0x2a0^M
[    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
kthread+0x1a8/0x1d0^M
[    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]

ret_from_kernel_thread+0x5c/0x64^M

But when I invoked [5] by hand, the bug did not show again [6].

[4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
[5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
[6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log

I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it.

Sorry for the inconvenience that I bring

Quoted 7 lines
Quoted 2 lines

Thanks
Zhouyi

Hi the above kernel null pointer dereference bug has nothing to do with Joel's fix because I can reproduce it on original 5.15.y [7] using as while loop [8] (after 36 iterations, the bug fires). So, Joel's patch is tested good!

Interesting. I have been running rcutorture's TREE03 on 5.15.y quite a lot and I don't see such an issue.

However, your logs showed the crash is SCSI related. These were the
recent SCSI commits in 5.15.y but I am not sure if it causes the issue:

Quoted 6 lines

13259b6 scsi: mpi3mr: Refer CONFIG_SCSI_MPI3MR in Makefile 513fdf0 scsi: ufs: Stop using the clock scaling lock in the error handler 7c26d21 scsi: ufs: core: WLUN suspend SSU/enter hibern

Perhaps report it to the scsi and/or stable lists, or do some web searches for if someone else sees it.

Thank Joel for your guidance!
I will continue the research following your instructions! I will report to you and the community once I have any process.

Thank you all!

Best Regards
Zhouyi

Quoted 2 lines

- Joel

Tested-by
#9 lore raw
Quoted 109 lines · Zhouyi Zhou
Quoted 107 lines · Joel Fernandes
Quoted 86 lines · Zhouyi Zhou
Quoted 84 lines · Zhouyi Zhou
Quoted 27 lines · Zhouyi Zhou
Quoted 25 lines · Paul E. McKenney
Quoted 22 lines · Joel Fernandes (Google)

For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures.

Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged.

[ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ]

For drivers/base/ portion:

Cc Frederic Weisbecker <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Zhouyi Zhou <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Fixes 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel")
Quoted 8 lines
Quoted 4 lines

Queued for further review and testing, thank you both!

It might be a few hours until it becomes publicly visible, but it will get there.

A new round of rcutorture test on fixed linux-5.15.y[3] has been performed in the PPC VM of Open Source Lab of Oregon State University [1], which will last about 29 hours. The test result on original linux-5.15.y is at [2].

From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-(

[    5.178733][    C1] BUG: Kernel NULL pointer dereference on read at
0x00000008
...
[    5.231013][    C1] [c00000001ff9fca0] [c0000000009ffbc8]
scsi_end_request+0xd8/0x1f0 (unreliable)^M
[    5.234961][    C1] [c00000001ff9fcf0] [c000000000a00e68]
scsi_io_completion+0x88/0x700^M
[    5.237863][    C1] [c00000001ff9fda0] [c0000000009f5028]
scsi_finish_command+0xe8/0x150^M
[    5.240089][    C1] [c00000001ff9fdf0] [c000000000a00c70]
scsi_complete+0x90/0x140^M
[    5.242481][    C1] [c00000001ff9fe20] [c0000000007e5170]
blk_complete_reqs+0x80/0xa0^M
[    5.245187][    C1] [c00000001ff9fe50] [c000000000f0b5d0]
__do_softirq+0x1e0/0x4e0^M
[    5.248479][    C1] [c00000001ff9ff90] [c0000000000170e8]
do_softirq_own_stack+0x48/0x60^M
[    5.250919][    C1] [c00000000a5e7c40] [c00000000a5e7c80]
0xc00000000a5e7c80^M
[    5.253792][    C1] [c00000000a5e7c70] [c0000000001534c0]
do_softirq+0xb0/0xc0^M
[    5.256824][    C1] [c00000000a5e7ca0] [c0000000001535ac]
__local_bh_enable_ip+0xdc/0x110^M
[    5.259414][    C1] [c00000000a5e7cc0] [c0000000001d75e8]
irq_forced_thread_fn+0xc8/0xf0^M
[    5.261921][    C1] [c00000000a5e7d00] [c0000000001d7ae4]
irq_thread+0x1b4/0x2a0^M
[    5.265298][    C1] [c00000000a5e7da0] [c00000000017d8c8]
kthread+0x1a8/0x1d0^M
[    5.269184][    C1] [c00000000a5e7e10] [c00000000000cee4]

ret_from_kernel_thread+0x5c/0x64^M

But when I invoked [5] by hand, the bug did not show again [6].

[4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log
[5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd
[6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log

I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it.

Sorry for the inconvenience that I bring

Quoted 7 lines
Quoted 2 lines

Thanks
Zhouyi

Hi the above kernel null pointer dereference bug has nothing to do with Joel's fix because I can reproduce it on original 5.15.y [7] using as while loop [8] (after 36 iterations, the bug fires). So, Joel's patch is tested good!

Interesting. I have been running rcutorture's TREE03 on 5.15.y quite a lot and I don't see such an issue.

However, your logs showed the crash is SCSI related. These were the
recent SCSI commits in 5.15.y but I am not sure if it causes the issue:

13259b6 scsi: mpi3mr: Refer CONFIG_SCSI_MPI3MR in Makefile 513fdf0 scsi: ufs: Stop using the clock scaling lock in the error handler 7c26d21 scsi: ufs: core: WLUN suspend SSU/enter hibern

Quoted 5 lines
Quoted 2 lines

Perhaps report it to the scsi and/or stable lists, or do some web searches for if someone else sees it.

Thank Joel for your guidance!
I will continue the research following your instructions! I will report to you and the community once I have any process.

Hi
I think I have chased down the reason behind the bug reported by SCSI subsystem following your guidance, and have reported to SCSI mainteners [1]
This is a fruitful journey, thank you all ;-)

So once again, Joel's patch is tested good ;-)

Thanks for your attention
Thanks
Zhouyi
[1] https://lore.kernel.org/lkml/CAABZP2ywCbu4Po63BDBE7U1WEqx4DF7F2CZjTqFp0dSDw-uziQ@mail.gmail.com/T/#u

Quoted 7 lines

Thank you all!

Best Regards
Zhouyi

Quoted 2 lines

- Joel