Skip to contentSkip to conversation

x86/paravirt: Guard against invalid cpu # in pv_vcpu_is_preempted()

PATCH In review Waiman Long 2 participants 9 messages Mar 25, 2019 – Apr 1, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/paravirt: Guard against invalid cpu # in pv_vcpu_is_preempted()

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +6 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }
 
Signature · 1 line
2.18.1
maintainer
#1 lore raw
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

Juergen

maintainer
#2 lore raw
In reply to Waiman Long
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

Juergen

#3 lore raw
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

Cheers,
Longman

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#4 lore raw
In reply to Juergen Gross
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

Cheers,
Longman

maintainer
#5 lore raw
Quoted 70 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

Quoted 7 lines

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

I think I'd prefer that.

Juergen

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

maintainer
#6 lore raw
In reply to Waiman Long
Quoted 70 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

Quoted 7 lines

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

I think I'd prefer that.

Juergen

#7 lore raw
Quoted 74 lines · Juergen Gross
Quoted 73 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Quoted 7 lines
Quoted 2 lines

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

I think I'd prefer that.

Juergen

It turns out that it may be caused by a software bug after all. You can ignore this patch for now.

Thanks,
Longman

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw
In reply to Juergen Gross
Quoted 74 lines · Juergen Gross
Quoted 73 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long

It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,

[    2.531077] Oops: 0000 [#1] SMP PTI
  :
[    2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20

To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.

---
 arch/x86/include/asm/paravirt.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +0 -0
index c25c38a05c1c..4cfb465dcde4 100644
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
 
Quoted 11 lines

arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681

Quoted 10 lines
arch/x86/include/asm/paravirt.h +6 -0
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
+	/*
+	 * Guard against invalid cpu number or the kernel might panic.
+	 */
+	if (WARN_ON_ONCE((unsigned long)cpu >= nr_cpu_ids))
+		return false;
+
 	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }

Can this really happen without being a programming error?

This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.

Quoted 2 lines

Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?

It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.

The race condition that I am looking at is as follows:

  CPU 0                         CPU 1
  -----                         -----
up_write:
  owner = NULL;
  <release-barrier>
  count = 0;

<rcu-free task structure>

                          rwsem_can_spin_on_owner:
                            rcu_read_lock();
                            read owner;
                              :
                            vcpu_is_preempted(owner->cpu); :
                            rcu_read_unlock()

When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.

Quoted 7 lines
Quoted 2 lines

Alternatively, I can also do the cpu check before calling vcpu_is_preempted().

I think I'd prefer that.

Juergen

It turns out that it may be caused by a software bug after all. You can ignore this patch for now.

Thanks,
Longman