Skip to contentSkip to conversation

clocksource/arm_arch_timer: fix a lockdep warning

PATCH New Qian Cai 6 participants 9 messages Dec 3, 2018 – Dec 14, 2018

Subsystems (3)
lore
Share & tools
lore

Qian Cai

author
#0 lore raw

PATCH v2 1/1 clocksource/arm_arch_timer: fix a lockdep warning

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

This is due to the commit cb538267ea1e ("jump_label/lockdep: Assert we hold the hotplug lock for _cpuslocked() operations").

Since it is applying a global workaround to all CPUs here, it did not hold any CPU locks in this path.

arch_timer_acpi_init
arch_timer_check_ool_workaround(ate_match_acpi_oem_info, table)
arch_timer_enable_workaround(wa, local = false)
for_each_possible_cpu()
per_cpu()

When hot-add a CPU, it will go with a slightly different route.

arch_timer_starting_cpu
__arch_timer_setup
arch_timer_check_ool_workaround(ate_match_local_cap_id, NULL)
arch_timer_enable_workaround(wa, local = true)
__this_cpu_write()

Hence, deal with them differently.

Fixes 450f9689f294 (clocksource/arm_arch_timer: Use

static_branch_enable_cpuslocked())

---

Changes since v1:
* Fixed the root cause instead of a workaround.

 drivers/clocksource/arm_arch_timer.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/clocksource/arm_arch_timer.c +9 -6
index 9a7d4dc00b6e..81dca7d31d13 100644
@@ -492,18 +492,21 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa
 
 	if (local) {
 		__this_cpu_write(timer_unstable_counter_workaround, wa);
+
+		/*
+		 * Use the locked version, as we're called from the CPU
+		 * hotplug framework. Otherwise, we end-up in
+		 * deadlock-land.
+		 */
+		static_branch_enable_cpuslocked(&arch_timer_read_ool_enabled);
 	} else {
 		for_each_possible_cpu(i)
 			per_cpu(timer_unstable_counter_workaround, i) = wa;
+		/* A global workaround is not on the CPU hotplug path. */
+		static_branch_enable(&arch_timer_read_ool_enabled);
 	}
 
 	/*
-	 * Use the locked version, as we're called from the CPU
-	 * hotplug framework. Otherwise, we end-up in deadlock-land.
-	 */
-	static_branch_enable_cpuslocked(&arch_timer_read_ool_enabled);
-
-	/*
 	 * Don't use the vdso fastpath if errata require using the
 	 * out-of-line counter accessor. We may change our mind pretty
 	 * late in the game (with a per-CPU erratum, for example), so
Signature · 7 lines
1.8.3.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#1 lore raw

PATCH RESEND v2 1/1 clocksource/arm_arch_timer: fix a lockdep warning

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

This is due to the commit cb538267ea1e ("jump_label/lockdep: Assert we hold the hotplug lock for _cpuslocked() operations").

Since it is applying a global workaround to all CPUs here, it did not hold any CPU locks in this path.

arch_timer_acpi_init
arch_timer_check_ool_workaround(ate_match_acpi_oem_info, table)
arch_timer_enable_workaround(wa, local = false)
for_each_possible_cpu()
per_cpu()

There is also another path did not have any CPU lock.

time_init
clocksource_probe
arch_timer_of_init
arch_timer_check_ool_workaround(ate_match_dt, np)
arch_timer_enable_workaround(wa, local = false)

When hot-adding a CPU, it will go with a slightly different route.

arch_timer_starting_cpu
__arch_timer_setup
arch_timer_check_ool_workaround(ate_match_local_cap_id, NULL)
arch_timer_enable_workaround(wa, local = true)
__this_cpu_write()

Hence, deal with them differently.

Fixes 450f9689f294 (clocksource/arm_arch_timer: Use

static_branch_enable_cpuslocked())

---

v2: fix the root cause instead of a workaround.

 drivers/clocksource/arm_arch_timer.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/clocksource/arm_arch_timer.c +9 -6
index 9a7d4dc00b6e..81dca7d31d13 100644
@@ -492,17 +492,20 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa
 
 	if (local) {
 		__this_cpu_write(timer_unstable_counter_workaround, wa);
+
+		/*
+		 * Use the locked version, as we're called from the CPU
+		 * hotplug framework. Otherwise, we end-up in
+		 * deadlock-land.
+		 */
+		static_branch_enable_cpuslocked(&arch_timer_read_ool_enabled);
 	} else {
 		for_each_possible_cpu(i)
 			per_cpu(timer_unstable_counter_workaround, i) = wa;
+		/* A global workaround is not on the CPU hotplug path. */
+		static_branch_enable(&arch_timer_read_ool_enabled);
 	}
 
-	/*
-	 * Use the locked version, as we're called from the CPU
-	 * hotplug framework. Otherwise, we end-up in deadlock-land.
-	 */
-	static_branch_enable_cpuslocked(&arch_timer_read_ool_enabled);
-
 	/*
 	 * Don't use the vdso fastpath if errata require using the
 	 * out-of-line counter accessor. We may change our mind pretty
Signature · 1 line
2.17.2 (Apple Git-113)
maintainer
#2 lore raw
Quoted 12 lines · Qian Cai

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

It seems to me we want something like:

---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +2 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }
 

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#3 lore raw

Hi,

On 10/12/2018 14:07, Peter Zijlstra wrote:
[...]

Quoted 9 lines
---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +0 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644

kernel/cpu.c void cpus_write_unlock(void) · line 320

Quoted 8 lines
kernel/cpu.c +2 -0
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }
 

Kinda hijacking the thread here - is that something we'd want to replace

40fa3780bac2 ("sched/core: Take the hotplug lock in sched_init_smp()")

with?

maintainer
#4 lore raw
Quoted 24 lines · Valentin Schneider

Hi,

On 10/12/2018 14:07, Peter Zijlstra wrote:
[...]

Quoted 9 lines
---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +0 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644

kernel/cpu.c void cpus_write_unlock(void) · line 320

Quoted 8 lines
kernel/cpu.c +2 -0
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }
 

Kinda hijacking the thread here - is that something we'd want to replace

Quoted 4 lines

40fa3780bac2 ("sched/core: Take the hotplug lock in sched_init_smp()")

with?

Possibly; and I have vague memories of other people running into this same thing too.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Tested-by
#5 lore raw
In reply to Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Qian Cai

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4
Quoted 1 line

It seems to me we want something like:

Quoted 19 lines
---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +2 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }
 
#6 lore raw
In reply to Peter Zijlstra
Quoted 25 lines · Peter Zijlstra
Quoted 12 lines · Qian Cai

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

It seems to me we want something like:

---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +0 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644

kernel/cpu.c void cpus_write_unlock(void) · line 319

Quoted 7 lines
kernel/cpu.c +2 -0
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }

Hmm, no. SYSTEM_SCHEDULING is what you want because RUNNING is set way too late.

Thanks,

tglx

#7 lore raw
Quoted 27 lines · Thomas Gleixner
Quoted 26 lines · Peter Zijlstra
Quoted 12 lines · Qian Cai

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

It seems to me we want something like:

---
 kernel/cpu.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/cpu.c +0 -0
index 91d5c38eb7e5..e1ee8caf28b5 100644
@@ -313,6 +313,8 @@ void cpus_write_unlock(void)
 
Quoted 9 lines

kernel/cpu.c void cpus_write_unlock(void) · line 319

Quoted 6 lines
kernel/cpu.c +2 -0
 void lockdep_assert_cpus_held(void)
 {
+	if (system_state < SYSTEM_RUNNING)
+		return;
 	percpu_rwsem_assert_held(&cpu_hotplug_lock);
 }

Hmm, no. SYSTEM_SCHEDULING is what you want because RUNNING is set way too late.

SYSTEM_SCHEDULING works well here too.

#8 lore raw
In reply to Qian Cai
Quoted 58 lines · Qian Cai

Booting this Huawei TaiShan 2280 arm64 server generated this lockdep warning.

[    0.000000]  lockdep_assert_cpus_held+0x50/0x60
[    0.000000]  static_key_enable_cpuslocked+0x30/0xe8
[    0.000000]  arch_timer_check_ool_workaround+0x128/0x2d0
[    0.000000]  arch_timer_acpi_init+0x274/0x6ac
[    0.000000]  acpi_table_parse+0x1ac/0x218
[    0.000000]  __acpi_probe_device_table+0x164/0x1ec
[    0.000000]  timer_probe+0x1bc/0x254
[    0.000000]  time_init+0x44/0x98
[    0.000000]  start_kernel+0x4ec/0x7d4

This is due to the commit cb538267ea1e ("jump_label/lockdep: Assert we hold the hotplug lock for _cpuslocked() operations").

Since it is applying a global workaround to all CPUs here, it did not hold any CPU locks in this path.

arch_timer_acpi_init
arch_timer_check_ool_workaround(ate_match_acpi_oem_info, table)
arch_timer_enable_workaround(wa, local = false)
for_each_possible_cpu()
per_cpu()

There is also another path did not have any CPU lock.

time_init
clocksource_probe
arch_timer_of_init
arch_timer_check_ool_workaround(ate_match_dt, np)
arch_timer_enable_workaround(wa, local = false)

When hot-adding a CPU, it will go with a slightly different route.

arch_timer_starting_cpu
__arch_timer_setup
arch_timer_check_ool_workaround(ate_match_local_cap_id, NULL)
arch_timer_enable_workaround(wa, local = true)
__this_cpu_write()

Hence, deal with them differently.

Fixes 450f9689f294 (clocksource/arm_arch_timer: Use

static_branch_enable_cpuslocked())

---

v2: fix the root cause instead of a workaround.

 drivers/clocksource/arm_arch_timer.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/clocksource/arm_arch_timer.c +0 -0
index 9a7d4dc00b6e..81dca7d31d13 100644

drivers/clocksource/arm_arch_timer.c void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa · line 501

Quoted 10 lines
drivers/clocksource/arm_arch_timer.c +7 -0
@@ -492,17 +492,20 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa
 
 	if (local) {
 		__this_cpu_write(timer_unstable_counter_workaround, wa);
+
+		/*
+		 * Use the locked version, as we're called from the CPU
+		 * hotplug framework. Otherwise, we end-up in
+		 * deadlock-land.
+		 */
+		static_branch_enable_cpuslocked(&arch_timer_read_ool_enabled);

I have the ugly feeling that it breaks the (equally ugly) big-little stuff where the boot CPU is affected. In this context, you'd end-up without the lock being taken and you'll get the same splat.

We could start testing the CPU number, but Peter's approach seem much more palatable to me.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...