Skip to contentSkip to conversation

xen: avoid crash in disable_hotplug_cpu

PATCH Superseded Olaf Hering 3 participants 11 messages Sep 6, 2018 – Sep 7, 2018 Xen hypervisor interface

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xen: avoid crash in disable_hotplug_cpu

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
 handle_vcpu_hotplug_event+0xb5/0xc0
 xenwatch_thread+0x80/0x140
 ? wait_woken+0x80/0x80
 kthread+0x112/0x130
 ? kthread_create_worker_on_cpu+0x40/0x40
 ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false for a cpu that was just removed.

---
 drivers/xen/cpu_hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/xen/cpu_hotplug.c +2 -0
index d4265c8ebb22..bf1e41ed9d41 100644
@@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
 
 static void disable_hotplug_cpu(int cpu)
 {
+	if (!cpu_is_hotpluggable(cpu))
+		return;
 	if (cpu_online(cpu)) {
 		lock_device_hotplug();
 		device_offline(get_cpu_device(cpu));
#1 lore raw
Quoted 44 lines · Olaf Hering

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
 handle_vcpu_hotplug_event+0xb5/0xc0
 xenwatch_thread+0x80/0x140
 ? wait_woken+0x80/0x80
 kthread+0x112/0x130
 ? kthread_create_worker_on_cpu+0x40/0x40
 ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false for a cpu that was just removed.

---
 drivers/xen/cpu_hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/xen/cpu_hotplug.c +0 -0
index d4265c8ebb22..bf1e41ed9d41 100644

drivers/xen/cpu_hotplug.c static void enable_hotplug_cpu(int cpu) · line 22

Quoted 4 lines
drivers/xen/cpu_hotplug.c +1 -0
@@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
 
 static void disable_hotplug_cpu(int cpu)
 {
+	if (!cpu_is_hotpluggable(cpu))

I think we should check both this and num_online_cpus() != 0.

Even though I don't believe cpu0_hotpluggable currently works, at some point it might.

-boris

drivers/xen/cpu_hotplug.c static void enable_hotplug_cpu(int cpu) · line 26

Quoted 4 lines
drivers/xen/cpu_hotplug.c +1 -0
+		return;
 	if (cpu_online(cpu)) {
 		lock_device_hotplug();
 		device_offline(get_cpu_device(cpu));

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
In reply to Olaf Hering
Quoted 44 lines · Olaf Hering

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
 handle_vcpu_hotplug_event+0xb5/0xc0
 xenwatch_thread+0x80/0x140
 ? wait_woken+0x80/0x80
 kthread+0x112/0x130
 ? kthread_create_worker_on_cpu+0x40/0x40
 ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false for a cpu that was just removed.

---
 drivers/xen/cpu_hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/xen/cpu_hotplug.c +0 -0
index d4265c8ebb22..bf1e41ed9d41 100644

drivers/xen/cpu_hotplug.c static void enable_hotplug_cpu(int cpu) · line 22

Quoted 4 lines
drivers/xen/cpu_hotplug.c +1 -0
@@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
 
 static void disable_hotplug_cpu(int cpu)
 {
+	if (!cpu_is_hotpluggable(cpu))

I think we should check both this and num_online_cpus() != 0.

Even though I don't believe cpu0_hotpluggable currently works, at some point it might.

-boris

Quoted 4 lines
drivers/xen/cpu_hotplug.c +1 -0
+		return;
 	if (cpu_online(cpu)) {
 		lock_device_hotplug();
 		device_offline(get_cpu_device(cpu));
#3 lore raw

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 9 lines · Olaf Hering
Quoted 6 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Olaf

#4 lore raw
Quoted 4 lines · Olaf Hering

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 1 line · Olaf Hering
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

And maybe that needs to be part of the check, in addition to cpu_is_hotpluggable() test.

Offlining CPU0 is problematic. For example, look at xen_pv_cpu_disable().

-boris

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
In reply to Olaf Hering
Quoted 4 lines · Olaf Hering

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 1 line · Olaf Hering
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

And maybe that needs to be part of the check, in addition to cpu_is_hotpluggable() test.

Offlining CPU0 is problematic. For example, look at xen_pv_cpu_disable().

-boris

maintainer
#6 lore raw
In reply to Olaf Hering
Quoted 11 lines · Olaf Hering

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 7 lines · Olaf Hering
Quoted 6 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

Quoted 4 lines
Quoted 1 line

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

We should add the Xen PV guest test to arch_register_cpu() and switch cpu0_hotpluggable off.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer
#7 lore raw
In reply to Olaf Hering
Quoted 11 lines · Olaf Hering

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 7 lines · Olaf Hering
Quoted 6 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

Quoted 4 lines
Quoted 1 line

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

We should add the Xen PV guest test to arch_register_cpu() and switch cpu0_hotpluggable off.

Juergen

#8 lore raw
In reply to Olaf Hering

Am Thu, 6 Sep 2018 22:31:45 +0200
schrieb Olaf Hering <[email protected]>:

Quoted 1 line

IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Another option is to do cpu_online() twice to check if device_offline did anything. Not sure if the compiler would fold the two checks into a single check.

Olaf

#9 lore raw
In reply to Olaf Hering

Am Thu, 6 Sep 2018 22:31:45 +0200
schrieb Olaf Hering <[email protected]>:

Quoted 1 line

IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Another option is to do cpu_online() twice to check if device_offline did anything. Not sure if the compiler would fold the two checks into a single check.

Olaf

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw
In reply to Boris Ostrovsky

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <[email protected]>:

Quoted 9 lines · Olaf Hering
Quoted 6 lines

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

This happens because handle_vcpu_hotplug_event is called twice. In the first iteration cpu_present is still true, in the second iteration cpu_present is false which causes get_cpu_device to return NULL. In case of cpu#0, cpu_online is apparently always true.

I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline. IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Olaf

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel