smp: Do not warn if smp_call_function_single() is doing a self call.
PATCH 1/1 smp: Do not warn if smp_call_function_single() is doing a self call.
If smp_call_function_single() is calling the function for itself, it's safe to run with irqs_disabled() == true.
I hit the warning because I'm in the below path in the .suspend callback of a "syscore_ops" to support hibernation for a VM running on Hyper-V:
hv_synic_cleanup() ->
clockevents_unbind_device() ->
clockevents_unbind() ->
smp_call_function_single().When the .suspend callback runs, only CPU0 is online and irqs_disabled() is true.
--- kernel/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/smp.c +1 -1
@@ -288,7 +288,7 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,* can't happen. */ WARN_ON_ONCE (cpu_online (this_cpu) &&irqs_disabled () - && !oops_in_progress); + && cpu !=smp_processor_id () && !oops_in_progress); csd = &csd_stack;if (!wait) {
Signature · 1 line
2.19.1
Quoted 11 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().
I'd suggest fixing clockevents_unbind() instead, something like (completely untested):
kernel/time/clockevents.c +10 -1
@@ -418,8 +418,17 @@ static void __clockevents_unbind(void *arg)static int clockevents_unbind (struct clock_event_device *ced,int cpu) {struct ce_unbind cu = { .ce = ced, .res = -ENODEV }; +int this_cpu; + + this_cpu =get_cpu (); + +if (cpu != this_cpu) +smp_call_function_single (cpu, __clockevents_unbind, &cu,1 ); +else +__clockevents_unbind (&cu); + +put_cpu (); -smp_call_function_single (cpu, __clockevents_unbind, &cu,1 );return cu.res; }
Quoted 13 lines
When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.Signed-off-by Dexuan Cui <[email protected]>--- kernel/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/smp.c +0
-0
kernel/smp.c int smp_call_function_single(int cpu, smp_call_func_t func, void *info, · line 291
Quoted 5 lines
kernel/smp.c +1
-1@@ -288,7 +288,7 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,* can't happen. */ WARN_ON_ONCE (cpu_online (this_cpu) &&irqs_disabled () - && !oops_in_progress); + && cpu !=smp_processor_id () && !oops_in_progress);
You already have 'this_cpu', no need to call smp_processor_id().
Quoted 3 lines
kernel/smp.c +0
-0csd = &csd_stack;if (!wait) {
Signature · 1 line
Vitaly
Quoted 6 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:
Quoted 7 lines
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.
Pray tell, how well do you think mutex_lock() works with interrupts disabled?
Quoted 16 lines
From: Peter Zijlstra <[email protected]>
Sent: Monday, April 15, 2019 5:21 AM
To: Dexuan Cui <[email protected]>Quoted 11 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().
Quoted 5 lines
Quoted 2 lines
When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.Pray tell, how well do you think
mutex_lock()works with interrupts disabled?
Good point. I realized generally speaking this patch makes no sense, so let me try the solution proposed by Vitaly, i.e. fix clockevents_unbind() instead.
Thanks,
-- Dexuan
Quoted 16 lines · Dexuan Cui
Quoted 16 lines
From: Peter Zijlstra <[email protected]>
Sent: Monday, April 15, 2019 5:21 AM
To: Dexuan Cui <[email protected]>Quoted 11 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.Pray tell, how well do you think
mutex_lock()works with interrupts disabled?Good point. I realized generally speaking this patch makes no sense, so let me try the solution proposed by Vitaly, i.e. fix
clockevents_unbind()instead.
That's still not the problem. You're calling clockevents_unbind_device() with IRQs disabled, that's not correct. It doesn't matter what clockevents_unbind() does thereafter.
You simply cannot do any of this with IRQs disabled, end of story.
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Dexuan Cui
Quoted 21 lines
From: Peter Zijlstra <[email protected]>
Sent: Monday, April 15, 2019 5:21 AM
To: Dexuan Cui <[email protected]>Quoted 6 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:Quoted 7 lines
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.Pray tell, how well do you think
mutex_lock()works with interrupts disabled?
Quoted 7 lines
Quoted 2 lines
Good point. I realized generally speaking this patch makes no sense, so let me try the solution proposed by Vitaly, i.e. fix
clockevents_unbind()instead.That's still not the problem. You're calling
clockevents_unbind_device()with IRQs disabled, that's not correct. It doesn't matter whatclockevents_unbind()does thereafter.
True. And before we start digging deeper into this, let's step back: why do we need to do clockevents_unbind_device() on hybernation? Can we just disable the device and re-enable it back on resume?
Actually, all usages of clockevents_unbind_device() in kernel are limited to Hyper-V and with Michael's patches moving this out of VMBus driver I think it can go away completely.
Signature · 1 line
Vitaly
Quoted 28 lines · Vitaly Kuznetsov
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Dexuan Cui
Quoted 16 lines
From: Peter Zijlstra <[email protected]>
Sent: Monday, April 15, 2019 5:21 AM
To: Dexuan Cui <[email protected]>Quoted 11 lines · Dexuan Cui
If
smp_call_function_single()is calling the function for itself, it's safe to run withirqs_disabled()== true.I hit the warning because I'm in the below path in the .suspend callback of a "
syscore_ops" to support hibernation for a VM running on Hyper-V:
hv_synic_cleanup()->clockevents_unbind_device() -> clockevents_unbind() -> smp_call_function_single().Quoted 5 lines
Quoted 2 lines
When the .suspend callback runs, only CPU0 is online and
irqs_disabled()is true.Pray tell, how well do you think
mutex_lock()works with interrupts disabled?Good point. I realized generally speaking this patch makes no sense, so let me try the solution proposed by Vitaly, i.e. fix
clockevents_unbind()instead.
Quoted 8 lines
Quoted 4 lines
That's still not the problem. You're calling
clockevents_unbind_device()with IRQs disabled, that's not correct. It doesn't matter whatclockevents_unbind()does thereafter.True. And before we start digging deeper into this, let's step back: why do we need to do
clockevents_unbind_device()on hybernation? Can we just disable the device and re-enable it back on resume?
Yes. That's the right thing to do. Simple solution is to implement the suspend/resume callbacks on the clock events device and be done with it.
Quoted 3 lines
Actually, all usages of
clockevents_unbind_device()in kernel are limited to Hyper-V and with Michael's patches moving this out of VMBus driver I think it can go away completely.
Correct. There was a driver which required that, but that's gone by now and of course nobody noticed that it was the last user. The reason why this exists was to allow switching out an active clocksource similar to the sysfs unbind file but without user space interaction.
Thanks,
tglx
Quoted 6 lines
From: Thomas Gleixner <[email protected]>
Sent: Tuesday, April 16, 2019 1:13 PMQuoted 4 lines
...
True. And before we start digging deeper into this, let's step back: why do we need to doclockevents_unbind_device()on hybernation? Can we just disable the device and re-enable it back on resume?
We do clockevents_unbind_device as part of hv_synic_cleanup(), which is called as a CPU hotplug callback: see vmbus_bus_init():
ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "hyperv/vmbus:online",
hv_synic_init, hv_synic_cleanup);
Yes, it looks the right thing is to implement the suspend/resume callbacks of the clock_event_device. Thank you for the suggestion! I'll look into this.
Quoted 2 lines
Yes. That's the right thing to do. Simple solution is to implement the suspend/resume callbacks on the clock events device and be done with it.
Agreed.
Quoted 3 lines
Quoted 3 lines
Actually, all usages of
clockevents_unbind_device()in kernel are limited to Hyper-V and with Michael's patches moving this out of VMBus driver I think it can go away completely.
Thanks for the heads-up! I'll rebase to Michael's patches.
Quoted 6 lines
Correct. There was a driver which required that, but that's gone by now and of course nobody noticed that it was the last user. The reason why this exists was to allow switching out an active clocksource similar to the sysfs unbind file but without user space interaction.
tglx
Thanks for the background sharing!
- Dexuan