Skip to contentSkip to conversation

Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

PATCH Superseded Vitaly Kuznetsov 4 participants 7 messages May 19, 2015 – May 20, 2015

Subsystems (2)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

Export cpu_hotplug_enable/cpu_hotplug_disable functions from cpu.c and use them instead of altering the smp_ops.cpu_disable in Hyper-V vmbus module.

Vitaly Kuznetsov (2):
cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable
Drivers: hv: vmbus: use cpu_hotplug_enable/disable

 drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
 kernel/cpu.c           |  3 ++-
 2 files changed, 6 insertions(+), 35 deletions(-)
Signature · 1 line
1.9.3
#1 lore raw

PATCH 1/2 cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

Loaded Hyper-V module will use these functions to disable CPU hotplug under certain circumstances.

---
 kernel/cpu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/cpu.c +2 -1
index 94bbe46..dc005e7 100644
@@ -193,6 +193,7 @@ void cpu_hotplug_disable(void)
 	cpu_hotplug_disabled = 1;
 	cpu_maps_update_done();
 }
+EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
 
 void cpu_hotplug_enable(void)
 {
@@ -200,7 +201,7 @@ void cpu_hotplug_enable(void)
 	cpu_hotplug_disabled = 0;
 	cpu_maps_update_done();
 }
-
+EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */
Signature · 1 line
1.9.3
#2 lore raw

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

2015-05-19 14:15+0200, Vitaly Kuznetsov:

Quoted 13 lines

Loaded Hyper-V module will use these functions to disable CPU hotplug under certain circumstances.

---
 kernel/cpu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/cpu.c +0 -0
index 94bbe46..dc005e7 100644

kernel/cpu.c void cpu_hotplug_disable(void) · line 193

Quoted 1 line
kernel/cpu.c +0 -0
@@ -193,6 +193,7 @@ void cpu_hotplug_disable(void)
 	cpu_hotplug_disabled = 1;

(I think it would be safer to make this into a counter, at least, so

  cpu_hotplug_disable() |
                        | cpu_hotplug_disable()
  cpu_hotplug_enable()  |

wouldn't end up with enabled hotplug, when this switch is intended for hotplug-and-die situations.)

Quoted 18 lines
kernel/cpu.c +2 -1
 	cpu_maps_update_done();
 }
+EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
 
 void cpu_hotplug_enable(void)
 {
@@ -200,7 +201,7 @@ void cpu_hotplug_enable(void)
 	cpu_hotplug_disabled = 0;
 	cpu_maps_update_done();
 }
-
+EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */
Signature · 1 line
1.9.3
maintainer
#3 lore raw

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

In reply to Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov

Loaded Hyper-V module will use these functions to disable CPU hotplug under certain circumstances.

What's wrong with get_online_cpus() ?

#4 lore raw

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

Quoted 5 lines · Peter Zijlstra
Quoted 2 lines · Vitaly Kuznetsov

Loaded Hyper-V module will use these functions to disable CPU hotplug under certain circumstances.

What's wrong with get_online_cpus() ?

We need to disable cpu offlining permanently (till the module is unloaded but this is highly unlikely). If we do get_online_cpus() instead we get

# echo 0 > /sys/devices/system/cpu/cpu1/online 
<nothing happens, pemanent hang followed by the following:>
[  240.247109] INFO: task bash:972 blocked for more than 120 seconds.
[  240.248937]       Not tainted 4.1.0-rc3_bug1167380_test_rkr+ #602
[  240.250515] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  240.252379] bash            D ffff880021b83c18     0   972    939
0x00000004
[  240.255068]  ffff880021b83c18 0000000000000001 0000000000000046
ffff880021b83be8
[  240.258140]  ffffffff81fdcdb0 ffff880037dbcfc0 ffff880037203bd0
ffffffff81fdcdb8
[  240.261275]  ffff880021b84000 0000000000000001 0000000000000000
0000000000000001
[  240.264402] Call Trace:
[  240.265423]  [<ffffffff81aecf77>] schedule+0x37/0x90
[  240.266748]  [<ffffffff81081872>] cpu_hotplug_begin+0xd2/0xe0
[  240.268210]  [<ffffffff810817a5>] ? cpu_hotplug_begin+0x5/0xe0
[  240.269675]  [<ffffffff810cb860>] ? prepare_to_wait_event+0xf0/0xf0
[  240.271285]  [<ffffffff81ad561a>] _cpu_down+0x8a/0x300
[  240.275294]  [<ffffffff81ad58c6>] cpu_down+0x36/0x50
[  240.276680]  [<ffffffff8160f954>] cpu_subsys_offline+0x14/0x20
[  240.278145]  [<ffffffff8160a505>] device_offline+0xa5/0xd0
[  240.279618]  [<ffffffff8160a620>] online_store+0x50/0xa0
[  240.280991]  [<ffffffff81607678>] dev_attr_store+0x18/0x30

...

And if we do the cpu_hotplug_disable() call we get:

# echo 0 > /sys/devices/system/cpu/cpu1/online 
-bash: echo: write error: Device or resource busy

This behavior is preferable.

Signature · 1 line
  Vitaly
#5 lore raw

PATCH 2/2 Drivers: hv: vmbus: use cpu_hotplug_enable/disable

In reply to Vitaly Kuznetsov

Commit e513229b4c38 ("Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors") was alteringd smp_ops.cpu_disable to prevent CPU offlining. We can bo better by using cpu_hotplug_enable/disable functions instead of such hard-coding.

---
 drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
 1 file changed, 4 insertions(+), 34 deletions(-)
drivers/hv/vmbus_drv.c +4 -34
index c85235e..e916ba2 100644
@@ -763,38 +763,6 @@ static void vmbus_isr(void)
 	}
 }
 
-#ifdef CONFIG_HOTPLUG_CPU
-static int hyperv_cpu_disable(void)
-{
-	return -ENOSYS;
-}
-
-static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
-{
-	static void *previous_cpu_disable;
-
-	/*
-	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
-	 * ...) is not supported at this moment as channel interrupts are
-	 * distributed across all of them.
-	 */
-
-	if ((vmbus_proto_version == VERSION_WS2008) ||
-	    (vmbus_proto_version == VERSION_WIN7))
-		return;
-
-	if (vmbus_loaded) {
-		previous_cpu_disable = smp_ops.cpu_disable;
-		smp_ops.cpu_disable = hyperv_cpu_disable;
-		pr_notice("CPU offlining is not supported by hypervisor\n");
-	} else if (previous_cpu_disable)
-		smp_ops.cpu_disable = previous_cpu_disable;
-}
-#else
-static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
-{
-}
-#endif
 
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
@@ -836,7 +804,8 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
-	hv_cpu_hotplug_quirk(true);
+	if (vmbus_proto_version > VERSION_WIN7)
+		cpu_hotplug_disable();
 
 	/*
 	 * Only register if the crash MSRs are available
@@ -1103,7 +1072,8 @@ static void __exit vmbus_exit(void)
 	for_each_online_cpu(cpu)
 		smp_call_function_single(cpu, hv_synic_cleanup, NULL, 1);
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
-	hv_cpu_hotplug_quirk(false);
+	if (vmbus_proto_version > VERSION_WIN7)
+		cpu_hotplug_enable();
 	vmbus_disconnect();
 }
 
Signature · 1 line
1.9.3
#6 lore raw
In reply to Vitaly Kuznetsov
Quoted 17 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Tuesday, May 19, 2015 5:16 AM
To: [email protected]
Cc: KY Srinivasan; Haiyang Zhang; [email protected]; Dexuan Cui; Ingo Molnar; Paul E. McKenney; Rafael J. Wysocki; Peter Zijlstra; Thomas Gleixner; Radim Krčmář
Subject: [PATCH 0/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

Export cpu_hotplug_enable/cpu_hotplug_disable functions from cpu.c and use
them instead of altering the smp_ops.cpu_disable in Hyper-V vmbus module.

Quoted 7 lines

Vitaly Kuznetsov (2):
cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable
Drivers: hv: vmbus: use cpu_hotplug_enable/disable

 drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
 kernel/cpu.c           |  3 ++-
 2 files changed, 6 insertions(+), 35 deletions(-)

Greg,

Can you take both these patches through your tree.

Regards,

K. Y

Quoted 3 lines

--
1.9.3

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�